By adopting the quality software processes of IEC 62304, companies are better able to develop a safe product, avoid expensive recalls, and ensure that the same development process underpins the maintenance and upgrade process.
The use of sophisticated medical devices now more than ever helps medical practitioners diagnose with ease and accuracy. Their level of dependency on devices, however, has raised concerns about ensuring the safety and quality of the devices.
Notably, medical devices rely heavily on third-party and legacy software, often referred to as software of unknown pedigree (SOUP). This SOUP forms the basis of new developments, which may now be subject to new medical device requirements or modern coding standards imposed by government, client demands or simply a policy of continuous improvement within the developer organization. The need to leverage the value of SOUP, while meeting new standards and further developing functionality, presents its own set of unique challenges.
An analysis of 3140 medical device recalls conducted between 1992 and 1998 by the FDA reveals that 242 of them (7.7%) are attributable to software failures. Of the software recalls, 192 (or 79%) were caused by software defects introduced after software upgrades. The high percentage of errors introduced during product upgrade has caused government agencies for medical devices to focus not only on development, but also on subsequent maintenance and the impact of software change on the existing system.
Because of this, many companies are changing their approach to improve their software processes as well as to adopt IEC 62304, a standard for design of medical products recently endorsed by the European Union and the United States. IEC 62304 introduces a risk-based compliance structure that ensures that medical applications comply with the standards suitable for their risk assessment. This compliance structure is classified into Class A through C where the failure of Class C software could result in death or serious injury.
IEC 62304 Software Development Lifecycle
IEC 62304 focuses on the software development process, defining the majority of the software development and verification activities. This process includes activities like software development planning, requirement analysis, architectural design, software design, unit implementation and verification, software integration and integration testing, system testing and finally software release.
This standard not only outlines requirements for each stage of the development lifecycle, but also takes care of the maintenance process, the impact of software change to the existing system, and the risk involved in implementing the software change. IEC 62304 also deals in detail with the effect of the SOUP items right from planning, requirement analysis, architectural design, maintenance and risk management phases.
IEC 62304 & SOUP
SOUP software that can be reused in the development of new devices has become prevalent since medical devices now tend to be built on general-purpose embedded hardware, along with an operating system, device drivers for USB, Ethernet, or graphics, file systems, network stacks, etc. The use of SOUP in medical devices has its advantage in that the manufacturer can concentrate on the application software.
Figure 1: Traditional functional testing can leave many parts of the code unproven. The blue line shows the parts of code exercised with traditional testing for correct functionality, the red dotted line demonstrates the parts of code that are used when the application runs in the field whereas the green dotted line demonstrates the code enhancements which are prone to call into service combinations of data previously unencountered, resulting in the possibility of tapping into those previously unexercised paths.
However, since the applications need to run device-specific functions, SOUP in medical devices adds challenges. Most SOUP modules are provided by third-party vendors, who don’t follow any software process, software standards or even document the code. And while it addresses platform challenges, SOUP is developed under stringent time schedules where the emphasis is on productivity, not standards compliance. When subjected to system tests that check functionality, SOUP projects typically show very poor code coverage, leaving many code paths unexercised. The blue curve in Figure 1 represents functionally tested code. When that code is deployed, the different data and circumstances are likely to use many untested paths for the first time, potentially creating unexpected results. The red dotted curve in Figure 1 illustrates the part of the code used when the application is run in the field.
The European System and Software Initiative “Performance of Errors through experience-driven Test” (PET) investigated this phenomenon and agreed that deployed code often possesses many errors. PET aimed to reduce the number of bugs reported after release by 50% and to reduce the hours of test effort per bug found by 40%. Interestingly, PET exceeded this, achieving 75% fewer reported bugs and 46% improvement in test efficiency. PET’s findings demonstrated that the number of bugs that can be found by newer methods of testing, such as static and dynamic analysis, is large, even if that code has been through functional system testing and subsequently been released.
The SOUP previously subjected to functional test is then deployed for further tests. Even if it worked well, some part of the code may not be exercised, even when the product is in the field. When SOUP code needs ongoing development for later revisions or new applications, previously unexercised code paths are likely to be called into use by combinations of data that had never previously been encountered, potentially creating unexpected results. The green dotted curve in Figure 1 illustrates the part of the code used when enhancements are made to SOUP code.
To counteract this potential weakness, a detailed structural coverage analysis needs to be done to ensure that there is no unexercised code in the legacy software. IEC 62304 mandates functional coverage and structural coverage of the legacy software along with a detailed analysis of the risk that could be introduced by the addition of such software. Functional coverage ensures that the software functions according to the system design requirements, while structural coverage ensures that all code sections are exercised and shown to work properly.
IEC 62304 requires that all SOUP items to be incorporated in the medical device design be identified along with the specification of their functional and performance requirements. The medical device manufacturer needs to ensure proper operation of any SOUP items and that they meet the functional and performance requirements.
The IEC 62304 software development process begins with software development planning, which includes a detailed plan on the SOUP items to be used. These details define how SOUP items are to be integrated within the existing system, how to manage the risk associated with the SOUP, and how software configuration and change management affect the system.
This is followed by software requirement management, architecture design, integration testing, system testing, risk management, maintenance and change management phase. At each phase of the software development lifecycle there is a need to maintain the traceability between all phases.
The traditional view of software development shows each phase flowing into the next, perhaps with feedback to earlier phases, and a surrounding framework of configuration management and process. Traceability is assumed to be part of the relationships between the phases. However, the mechanism by which trace links are recorded is seldom stated.
In reality, while each individual phase may be conducted efficiently thanks to investment in up-to-date tool technology, these tools seldom contribute automatically to any traceability between the phases. As a result, the links between them become increasingly poorly maintained over the duration of projects. The net result is absent or superficial cross checking between requirements and implementation and consequent inadequacies in the resulting system. To gain true automated traceability requires a Requirements Traceability Matrix (RTM). The RTM sits at the heart of every project and is a key deliverable within many development standards, including IEC 62304.
Requirements Traceability and SOUP
Figure 2: The requirements traceability matrix (RTM) plays a central role in a development lifecycle model even when SOUP items are part of the system. Artefacts at all stages are linked directly to requirements matrix and changes within each phase automatically update the RTM.
The Requirements Traceability Matrix—a widely accepted practice for managing and tracing requirements—plays a vital role in managing the software requirements as well as the SOUP items to be used in the system. RTM helps to establish traceability between the high-level requirements pertaining to SOUP with the architectural design of the medical device application (Figure 2).
To ensure that SOUP can meet the system-level requirements outlined by IEC 62304, the medical device manufacturer needs to specify:
- Functional and performance requirements for the SOUP item necessary for its intended use
- Manufacturer specifications for the system hardware and software necessary to support the proper operation of the SOUP item
- Details to verify that the medical device architecture supports proper operation of any SOUP items
In most cases, the SOUP items are delivered as source code but without design documents, which makes it difficult to analyze them. Use of modern test tools helps in visualizing the legacy code design.
The system visualization facilities provided by modern test tools are extremely powerful, whether applied to statement blocks, procedures (or classes), applications and/or system wide. The static call graph shown in Figure 3a depicts a hierarchical illustration of the application and system entities, and the static flow graph in Figure 3b shows the control flow across program blocks. The use of color-coded diagrams provides considerable benefit in understanding SOUP. Such call graphs and flow graphs are just part of the benefit of the comprehensive analysis of all parameters and data objects used in the code.
Requirement management and traceability have already proven their advantage in the software development lifecycle to ensure that all requirements are implemented and that all development artifacts can be traced back to one or more requirements. Similarly, requirement management and traceability ensure that SOUP items are added and verified based on system requirements.
Figure 3: The static call graph (a) and flow graph (b) represent the structure and the logic of the code in graphical form.
RTM provides traceability between the architectural design and the SOUP items. Since these items are delivered in source code and are now required to fulfill system-level testing for compliance to IEC 62304, it becomes the manufacturer’s responsibility to verify the code.
The sloppiness of most SOUP items adds stress to the requirement of rigorous verification and risk analysis for the system integrator. Because verifying SOUP is so time-consuming, developers typically address a subset of the coding standard initially, gradually adopting additional rules. While test tools only identify but do not correct the violation and latent errors present in the code, they do speed correction of code by pinpointing problem areas.
IEC 62304 expects the medical device manufacturer to evaluate the software anomaly lists published by the supplier of the SOUP item to determine if any of the known anomalies could create a sequence of events that could result in a hazardous situation.
The static analysis capability of the test tools identifies the anomalies and their impact on the software system. If any additional anomalies that are not part of the list published by the supplier of SOUP are identified, they should be conveyed to the respective vendor to address the problem.
After static analysis and correction of anomalies is complete, dynamic analysis, including system, integration and unit testing, is performed to verify the functional and structural coverage of the SOUP items. Although system-wide functional testing provides the functional overview of the SOUP items, it does not test all code paths. Test tools identify the exercised parts of the software highlighting the areas that require attention, and these areas are put through unit tests to ensure each unit functions correctly in isolation.
Running functional tests and structural coverage analysis makes sure all code paths are exercised and the interfaces between multiple units are verified. It also helps to ensure the system functions per the design, even with the integration of the SOUP items. Notably, IEC 62304 requires that verification of the SOUP items follows the integration plan made during software planning, again indicating the elevated emphasis IEC 62304 places on ensuring upgrades to medical software do not introduce errors.
RTM provides traceability between the various analyses performed on SOUP items against the test plan established earlier. This test plan contains test cases to be carried out and their expected results based on the system requirements. Using RTM, project managers can estimate the impact of SOUP items to be incorporated and how they affect the safety of the system.
Maintenance of SOUP Items
Many incidents in the medical device industry are related to service or maintenance of medical device systems including inappropriate software updates and upgrades. SOUP items also play a major role here since these items are delivered by different vendors and need to be verified.
In IEC 62304, the software maintenance process is considered as important as the software development process. This emphasis on maintenance aims at curbing the high percentage of medical device software defects introduced after product release, i.e., during software maintenance.
The maintenance process requires that the manufacturer monitor the feedback of the released product from both within the organization and from the user. This feedback must be documented and analyzed to determine whether a problem exists. When a problem is found, a problem report is created and analyzed to determine whether SOUP items contributed to the problem. If SOUP is a problem, the issue has to be conveyed to the respective vendor to address the problem with upgrades or patches.
IEC 62304 requires the manufacturer to establish procedures to evaluate and implement upgrades, bug fixes, patches and obsolescence of SOUP items. Each upgrade, bug fix and patch has to be analyzed and verified to determine whether additional potential causes are introduced by these upgrades contributing to a hazardous situation. As always, it is necessary to determine whether additional software risk control measures are required.
During maintenance, the manufacturer is required to analyze changes to the SOUP items to determine whether the software modification could interfere with the existing risk control measures. The manufacturer must establish a scheme for the unique identification of configuration items and their versions. For each SOUP configuration item used, the manufacturer needs to document the title, SOUP manufacturer name and unique SOUP designator. This identification identifies the software configuration items and the versions included in the medical device software.
LDRA
Wirral, UK.
+44 0151 649 9300.
[www.ldra.com].
