These days a continuous integration (CI) tool is much more than “nice to have” during software development. And when it comes to the design and development of a software medical device, good continuous integration practices are even more imperative. I say this as someone who has spent excessive amounts of time wading through documents and double checking references, checking traceability, versions and making sure design outputs are properly documented in a Design History File (DHF). CI shouldn’t be thought of as nice to have, rather, it is a key process in making software medical device manufacturing possible.

A CI tool can take the sometimes feeble attempts of humans to make large amounts of documentation consistently traceable and force the computer system to do what it does best—continuously. The use of a CI tool is not simply an esoteric practice for those who are fond of its incorporation. CI is an activity that successful development teams have always attempted, but they have too often failed to utilize software tools to ease the process. Going a step further, development teams can use a CI tool to simplify steps that they may never have dreamed of before!

Continuous Integration refers to the continuous compiling and building of a project tree as well as continuous testing, releasing and quality control. This means that throughout the project, at every stage, the development team will have a build available with at least partial documentation and testing included. In general, CI builds are performed in an environment that closely matches the actual production environment of the system. A CI environment should be used to provide statistical feedback on build performance, tests and incorporation of a version control system and ticketing systems. In a development environment, the team may use a version control tool to link to tickets. In this way, any CI build will be linked to a specific change set, thereby providing linkage to issues, requirements and, ultimately, the trace matrix.

CI builds should occur frequently enough that no window of additional version control update occurs between commit and build, and such that no errors can arise without developers noticing them and correcting them immediately. This means that for a project that is in development, it should be configured that a checking triggers a build in a timely manner. Likewise, it is generally a good practice for the developer committing a change set of code to verify that his or her own change set does not break the continuous integration build. There is little overhead to creating many CI builds. There is, however, potential downside to not performing CI builds frequently enough.

Most software engineers think of a build as the output of compiling and linking. I suggest moving away from this narrow definition and expanding it. A “build” is a completion (in both the compiler sense and beyond) of all things necessary for a successful product delivery. A CI tool runs whatever scripts the development team tells it to run. As such, the team is free to use the CI tool as a build manager. It can compile code, create an installer, bundle any and all documents, create release notes, run tests and alert team members about its progress.

Jenkins CI

For the purposes of this article, the focus will be on one specific CI tool, Jenkins CI. This is one of the more popular open source tools available. Jenkins CI, the continuation of a product formerly called Hudson, allows continuous integration builds in the following ways:

1. It integrates with popular build tools (ant, maven, make) so that it can run the appropriate build scripts to compile, test and package within an environment that closely matches what the production environment will be.

2. It integrates with version control tools, including Subversion, so that different projects can be set up depending on projection location within the trunk.

3. It can be configured to trigger builds automatically by time and/or change set (i.e., if a new change set is detected in the Subversion repository for the project, a new build is triggered).

4. It reports on build status. If the build is broken, it can be configured to alert individuals by email.

Figure 1 gives an example of what a main page for Jenkins CI (or any CI tool) may look like. It can be configured to allow logins at various levels and on a per-project basis. This main page lists all the projects that are currently active, along with a status (a few details about the build) and some configuration links on the side. These links may not be available to a general user.

Clicking any project (“job”) links to further details on the build history and status. This image provides us details on what the overview screen in the CI environment might look like, but it is at the detailed project level that we see the real benefit of packaging that can be performed by a well set up CI environment.

DHF Requirements

Medical device software is audited and controlled by standards defined by the FDA, specifically Code of Federal Regulation, Title 21, parts 11 and 820 (21 CFR). Many of the requirements laid out in this difficult-to-understand guidance can be made easy when we use a CI environment throughout the course of project design and development. Looking specifically at the quality system requirements laid out by 21 CFR Part 820.30, Subpart C – Design Controls, it becomes apparent that a good CI environment can help to address each. A major consideration, perhaps the major consideration, is the completeness of the Design History File (DHF).

Figure 1 The main page of a CI tool will allow the user to navigate to various projects and levels within projects. It will also establish a system of permission levels and allow users to quickly see information on project status.

The “or reference” part of this statement stands out. Traditionally, medical device manufacturers have thought of the DHF as a physical, self-contained item. But with a project of any complexity, it isn’t difficult to imagine how quickly a DHF may grow into an unruly mess of “stuff.” Why not simply leverage software tools to make the process seamless? Using a CI tool, development teams can pull together a baseline of all the elements of a DHF as frequently as they wish to. Furthermore, they can do so with a degree of accuracy that cannot be achieved through the diligent (yet distractible) legwork of a busy team.

I propose that the DHF need not be a single physical or soft folder with duplicate copies of items. Leveraging the CI environment along with the version control system, it is a much better idea to think of the DHF as a snapshot of all relevant design outputs at a given point in time. To that end, the development team can have many snapshots of the DHF throughout the project lifecycle. To achieve this, they need simply to define this process in their standard operating procedures and work instructions. To those who have only worked with a DHF as a particular folder with specific subsets of documentation within, this approach, while it makes sense, takes a bit of a leap from the traditional DHF mindset to attempt.

The CI can present much more information than a simple build history

Figure 2 shows what a (simple) project setup may provide in the way of such packaging. It is up to the team to determine how much or how little the CI handles, but it makes the most sense to allow it to do what computers do very well and what humans tend not to do as well: align things.

The sample in Figure 2 shows that more than simply the build package can be included in the CI output. Development teams can bundle other things that are required by the DHF, including any manuals, plans, requirements and so on. This environment is starting to look more and more like a DHF.

The CI Environment

The CI build server should closely mimic the environment in which the final product will be deployed. This helps establish a level of confidence with regard to system compatibility prior to user acceptance and integration testing. It must also have access (through the version control and/or ECM system) to all the design controls and documents necessary to build a complete DHF. It is generally best to use a single version control system for everything. It doesn’t make sense, for example, to store source code in one version control system and documents in another. Doing so makes importing of all necessary items difficult, if not impossible.

There are a number of benefits to utilization of a CI server during project design and development. Do not think of CI as a tool only for software builds. Integrated with the project version control system, it can serve as much more. Within the CI build history we now have a mechanism by which we can recreate the precise environment and moment in time in which the build was generated. From code to documentation, all design history (think DHF) can be traced to the build.

The CI build gives the development team prompt feedback on the build status. If compilation fails, tests fail or some requirement element cannot be packaged, the entire team is flagged immediately. To this end, the entire team will know that a particular check-in has broken something. This feedback will eliminate the fear that an unknown break could be so extensive that progress will come to a screeching halt. The near real-time feedback of the CI build saves valuable time (and stress!) throughout development and even design.

Project progress tracking (tickets, tests, etc.) gives the team and management the ability to see a quick overview of project trends, including completion of the project, test results, code quality, reviews, documentation and so on.

Along with project progress tracking, a major feedback loop is completed. Continuous integration provides continuous feedback for all team members. Even software peer reviews can be a part of this feedback loop, with every change set and/or ticket completion triggering a mini peer review. When a team reviews smaller portions of code by paying attention to every change set, we have more manageable reviews as well as improved team understanding of each other’s work.

Triggering Builds and Keeping Them Healthy

It is important for team members to focus on keeping the CI build in a “healthy” condition. CI builds can and do break for a number of reasons. This is to be expected.

As an example, we will use Jenkins-CI to automatically perform a CI build every hour if there is a change in the repository. The system will be configured to send emails to the development team if there is a problem with the build—i.e., if a change set breaks the CI build. It is anticipated that the CI build will break from time-to-time, however, a broken build should not be left unattended.

A common cause of a broken CI build is a lack of attention to the build script. Each developer is responsible for making certain that the ant build scripts are up to date with all required changes. We cannot rely on the build scripts that are generated by an IDE. There are certainly more possible causes that could be added to the above list. It is a good idea for each developer to trigger a CI build immediately following any Subversion commit to ensure that the CI build has not been broken. If a CI build continues to be broken without being addressed, the team leader and/or project manager may revert the offending change set and re-open any related issue.

Design history storage using CI environment

Using a CI Environment to Replace the Traditional DHF

Naturally, an important part of continuous integration is having a CI build that can be checked regularly for continued build success. This is probably what is commonly thought of as the key benefit, but there is much more to be gained. Any CI environment that is worth using will allow the team to incorporate packaging of key project items with each build. This includes important documents, tests (both manual and automated test outcomes can be packaged), requirements, design specifications and build results (deployment packages, libraries, executables, installers, etc.). The important thing to note here is the fact that, used wisely, the CI environment can provide a snapshot of all project outputs at any given point in time (Figure 3). Hopefully it is becoming clear that this gives us the possibility of automated DHF creation. Not only that, but we have a much more detailed DHF throughout the life of a project and not merely at a point in time in which a particular freeze was performed.

The DHF is much more than a loosely controlled folder with a number of documents shoved in. The DHF, when properly defined, is now all of the history that goes into the design and development of the software product, and the continuous integration environment is the glue that holds it together.

Matthew T. Rupert.

[www.matthewrupert.net].

Jenkins CI.

[www.jenkins-ci.org].

Medical devices, such as infusion pumps, patient monitors and MRI scanners, can be just as susceptible to malware as standard computers. Keeping them secure in any networked environment is certainly challenging, and the stakes are particularly high for these particular applications since they can affect patient care and outcomes.

Proving this point, McAfee and a medical equipment manufacturer recently raised awareness of security holes with potentially life or death consequences; they identified a networked insulin pump with a security flaw, which allows the device to be hacked and subsequently administer a potentially lethal amount of insulin to diabetes patients. Although not typically the target of cyber-attacks, medical equipment can become “collateral damage” in a malware outbreak, or even be the weak link that opens the door to a cyber-attack.

As the complexity of the network increases, securing devices becomes more complex for both the manufacturers and hospital IT organizations. However, this complexity is reduced significantly when medical devices are designed for security using models similar to typical networked clients. This synergy enables hospital IT personnel to apply consistent security strategies across the network, making it easier to administer and monitor equipment. Moreover, as new technologies and methods roll out to thwart attacks, they can be implemented in a similar fashion across the network.

There isn’t a single security solution capable of addressing all future risks; instead, most would agree it’s necessary to implement a series of different defenses across the system. This can be done using properly implemented layered security that enforces security policy from the CPU to the application software, as outlined here and demonstrated by the Intel Medical Security Reference Platform. In the best case, devices will be fully protected; and in the worst case, malware is detected faster, allowing counteractive action to be taken before any harm is done.

Device Security Challenges Today

One of the challenges facing hospital IT organizations is the large variety of hardware and software systems they must manage and secure. Further complicating matters, many equipment manufacturers develop unique security solutions, often as the result of designing purpose-built solutions based on non-standard or proprietary components. Consequently, it can be difficult to determine whether they comply with the security policies of the purchasing hospital and if they will be maintainable for the expected life of the devices.

Devices based on non-standard platforms may present other drawbacks, including the need to send them to the manufacturer for upgrades, security or otherwise, making them unavailable for a period of time. Additionally, it may be more difficult to capitalize on the latest advancements developed to secure IT infrastructure built with standards-based computing technology. For instance, hardware-assisted virtualization offers security benefits by providing an additional layer of security protection that complements software-only solutions.

It can also be challenging for organizations to reach consensus on security policy due to conflicting viewpoints and goals of key stakeholders. As an example, security officers tend to advocate locking down systems to better protect the network, while IT managers gravitate toward opening up the network to deliver the best end user experience. A mutually acceptable course may be found with a layered security model implemented on standards-based platforms, which will improve device security and lower hospital IT support requirements.

Like other devices on the network, once compromised, medical devices could be the vehicle for launching all sorts of attacks. They can be used to harm patients, access patient records, initiate network attacks—like denial of service (DoS)—or spread malware to other systems on the network. To stop such actions, it is necessary to prevent hackers and malware from breaching the platform. While the basic principle behind securing a platform is conceptually easy to understand, it is far more difficult to realize in practice.

The guiding principle is to protect the system by ensuring that any malware that somehow infiltrated a system cannot execute; if malware is present on the system, it cannot be allowed to embed itself in system memory. In reality, however, the most problematic malware finds a way to load itself into memory and obscure its presence; consequently, the platform’s security mechanisms are unable to discover it and take appropriate action.

Layered Security Model

Although there are no ironclad solutions, a layered security approach, with safeguards deployed throughout the platform, goes a long way toward providing robust protection against the vast majority of attacks. The basic premise is that by creating multiple barriers, a device has more opportunities to discover the malware before it causes harm, which forces hackers to write more sophisticated malware in order to circumvent all the lines of defense. Additionally, a well-designed layered defense helps contain malware, thus increasing the possibility that a device can continue to perform safety-critical tasks even when attacked.

Figure 1

Using a layered security model, Intel, Wind River and McAfee developed a secure platform for medical devices, demonstrated by the Intel Medical Security Reference Platform. This proof-of-concept incorporates eight security safeguards spanning multiple layers: hardware, virtualization, operating system and services software, as shown in Figure 1. The platform is designed with off-the-shelf components, and it applies security policy consistent with standard IT practices.

Eight Safeguards for Protecting Medical Devices

In healthcare, networked medical devices can fall victim to all types of perpetrators using a wide variety of methods. This section explores potential vulnerabilities and suggests safeguards, implemented across the platform, that either prevent attacks or minimize their impact until corrective action is taken.

Objective 1: Stop unauthorized data copying

Data is the life blood of the connected hospital, and it has to flow freely to add value. But how accessible can sensitive data be, and can it really be protected in a world of outsourcing, portable storage devices, Facebook and Twitter?

Objective 2: Prevent untrusted code execution

Medical devices, unlike tablets and laptops used by hospital staff, typically run a predetermined set of applications that are carefully controlled by the manufacturer. Two approaches for ensuring only the trusted applications can execute are called blacklisting and whitelisting. PC users are familiar with blacklisting from running anti-virus software that searches for bad software and neutralizes it. Whitelisting is a “lighter” approach and is well-suited for embedded devices running only known, trusted software; the permitted code is enumerated, and any application or file not on the list is prevented from executing.

Objective 3: Interrogate incoming packets

Viruses often gain access to medical devices through the network. This common method of attack can be curtailed by locking down access so only legitimate communications are received and transmitted by the device.

Objective 4: Protect data and communications

Once compromised, a medical device can become a base from which a hacker launches attacks on other devices and systems on the hospital network.

Objective 5: Prevent unintended interactions between applications

A hacker can infiltrate one application with the intention of using it to gain access to another application’s data. After malware embeds itself in system memory, it will look for software applications and files to exploit by accessing their memory space. To reduce the harm malware can cause, restrict the number of software elements it has access to, thus greatly limiting a virus’ ability to move around. This can be achieved using virtualization technology to run applications in their own secured partitions.

Objective 6: Prevent device performance degradation due to poorly functioning code

Wreaking as much havoc as a virus, a badly coded application or an inadequately tested patch can consume copious amounts of computing resources, and ultimately have the same effect as a DoS attack. Left to run on unchecked, poorly functioning code can take precious CPU cycles and memory away from a medical device’s safety-critical applications, whose performance may degrade to the point of putting the patient at risk.

Objective 7: Reduce attack surface

Viruses frequently enter devices via network ports, so controlling this exposure can minimize security vulnerabilities.

Objective 8: Harden device against unexpected failures

The software complexity of modern medical devices makes it nearly impossible to exhaustively test for all the possible ways in which a system can be compromised. Negative testing, using techniques such as Fuzz testing, can alleviate some of this risk.

As of today, no single security solution offers 100 percent protection. Living with this reality everyday, hospital IT organizations must sort through countless solutions and support a large number of them. The complexity is multiplied by purpose-built medical devices incorporating unique and sometimes obscure solutions, which increases support effort.

Security cannot be bolted on as an afterthought at the end of the development cycle. Addressing security concerns must be part of the design process—from an analysis of all attack vectors that might be used by a hacker, through the selection of secure building blocks, to thorough security-focused testing—which is made an integral part of the medical device release checklist.

Moving forward, medical devices using standards-based platforms based on IT infrastructure can greatly simplify security management while offering state-of-the-art security protection. Another important criterion for security architecture is its effectiveness over the typical lifespan of devices—typically 10-15 years; such resiliency is enhanced by the Intel, Wind River and McAfee layered security approach outlined in this article.

Intel

Santa Clara, CA.

(408) 765-8080.

[www.intel.com].

Wind River

Alameda, CA.

(510) 748-4100.

[www.windriver.com].

Not all people who are legally blind have completely lost their sight. In fact many retain enough vestibular functional vision to perform various tasks, provided the image entering their eye is sufficiently enhanced and optimized for their unique retinal and neurological visual processes. Using a combination of advanced hardware, advanced software and innovative design, these people can now see. eSight Corporation and RoweBots worked together to produce the ultimate pair of augmented imaging glasses—Alvios Intelligent Eyewear.

Alvios Intelligent Eyewear (Figure 1) incorporates a miniature ophthalmic quality video camera mounted at the bridge of the nose, a standalone processor to modify and enhance the video, and two tiny high-resolution video screens in front of the user’s eyes. By wearing Alvios in an elevated position, people with low vision can use their often healthy peripheral vision to easily capture objects of interest in the world around them. Then, once the wearer spots an object they wish to observe more carefully, simply glancing up into the computer image shows the same object with various video enhancements that stimulate what remains of their impaired central vision. Alternatively, the wearer can place the video screens in a more immersive orientation, viewing the computer generated images full time. This mode is better suited for more stationary activities such as reading, watching television, doing crafts, or viewing a presentation or the theater.

Each individual has a very unique remaining visual function and personal goals and activities they wish to pursue. Alvios is configured by trained optometrists and ophthalmologists to make the most of what remains of a person’s specific functional visual performance.

Figure 2 shows the physical layout of the intelligent eyewear. You can see the camera receiving an image from the outside world coupled into a unit with two tiny displays—one for each eye. This head-mounted unit connects to a belt-mounted unit with a bidirectional serial data stream. The belt-mounted unit provides the processing power, batteries and various control features. The video camera transmits the image to an FPGA located in the belt unit. The FPGA provides initial processing on the image in real time and relays the augmented image into the camera subsystem. The Unison RTOS is used from this point to set up a signal processing software pipeline to further transform the image and finally output it through the cable to the head-mounted display subsystem. The display subsystem uses additional signal processing in the FPGA to prepare the image for display. The data flows from the FPGA directly to the tiny screens, which the visually impaired person sees roughly 1 cm in front of his or her eyes.

The key signal processing functions are:

• Image capture
• Autofocus
• Zoom
• Color correction
• Image enhancement
• Image display

The FPGA performs the first and last functions: initial image capture and display. The remaining functions are performed on an OMAP processor under software control with special data acquisition and image processing hardware. The signal processing starts with the setup of the processing and transformation environment during initialization. The flash resident software eliminates the boot phase, speeding startup. The FPGA’s firmware loading and the various hardware components initialization follows this under control of the Unison OS. The Unison OS is the core software component that does the initialization and provides the key software elements for the underlying I/O and processing system. The Unison image driver supports a full range of functions and provides queued buffers for processing with a variety of algorithms. The zoom function allows a portion of an image to be selected and examined in greater detail inside the driver. Real-time image statistics allow the application to provide auto-focus, auto-contrast and automatic color correction by accessing special features of the image driver. The buffer queuing system provides minimal buffer copying during the signal processing functions. These buffers eventually end up directly in the display system for output through the FPGA to the micro displays. Maximization of the available processing power and minimization of bus bandwidth directly results from this approach, making it a significant design feature. The application software provides overall control to the user, setup and selection of the signal processing functions and overall coordination of the various application components. The user controls the device from a set of controls on the belt unit. The most commonly used commands can be easily selected by the user. Futhermore, the trained clinician can customize the user interface, providing access to those features that provide the most benefit, and hiding those inappropriate for the specific user.

For a person unable to see for many years, using Alvios Intelligent Eyewear can be a powerful experience. This unique and powerful solution allows blind people to see again and mitigates one of the most significant losses one can experience.

Critical Design Issues
Just like many other products, size, weight and power (SWAP) are critical in the design. Careful tradeoffs were made to maximize utility while minimizing power consumption and weight. One critical design issue that the team grappled with was the weight of the headset. Alvios needs to feel more like a pair of glasses rather than a head-mounted helmet. To enhance vision, careful adjustment is required for each patient and the equivalent of a pair of glasses needs to be included in the headset. Battery life directly affects usability in real life situations making it a critical function.

High-performance image processing provides the image transformations regarded as the realm of FPGAs and DSPs—neither particularly power miserly. Alvios Intelligent Eyewear needs to be used for hours without recharging. This type of processing and time duration precluded including the battery in the head-mounted portion of the system as it would be too heavy. Lithium Ion battery choices can reduce the size and weight of the battery, but the requirement for a significant amp hour rating remains. With the battery having significant weight, it needs to be mounted where the user can more easily tolerate carrying the weight. Originally Alvios utilized the Linux OS. The application code consisted of a multithreaded single process application. The main issues with Linux were the high complexity of drivers and driver development and the separation imposed by Linux between user space and kernel space. This involved a large amount of buffer management and copying between user space and kernel space. In addition, with Linux there was a lot of required maintenance with many specialized drivers and a moving set of kernel versions. Significant effort was required to integrate DSP functions and boot times were inconveniently lengthy.

Kim Rowe, President of Rowebots, wears ALIVIOSª Intelligent Eyewear

The complexity of the Linux environment required substantial full time resources to track releases and various specialized drivers. Often the effort of developing drivers and integrating them fell on the team, and as soon as they were done, new changes forced immediate redevelopment. The cost and delays of this approach were prohibitive.

Real-time, low latency performance is critical for Alvios users. Linux’s non-hard real-time performance creates a limitation as a solution in this instance; hard real-time performance creates an enhanced user experience. By getting more out of the same hardware, the user experience can be substantially enhanced without an increase in the bill of materials (BOM) cost. Seamless integration of DSP processing makes design, maintenance and system understanding much simpler. While DSP was partially integrated into the off-the-shelf Linux environment available with the OMAP, driver integration of these DSP functions would be required to achieve optimum performance. This amounted to significant additional work at the Linux driver level where undue complexity was already a huge burden.

The Software Solution
After carefully evaluating several alternatives, the decision to replace the Linux OS with the Unison OS from RoweBots proceeded as the low-risk and low-cost approach with the biggest long term benefits. Selecting the Unison OS platform for this application promised a number of advantages. The simplified OS architecture would make developing and maintaining specialized drivers much simpler. Given the real-time response of Unison and the elimination of substantial complexity and overhead, the performance was expected to improve. The drivers from Unison could replace the Linux drivers, offering identical functionality without any maintenance and support burden. In addition, DSP functions could be seamlessly integrated without penalty, without buffer copying and with an integrated debug solution. Unison features the capability for a flash-based image and nearly zero boot time. It was possible to rapidly port the application with only minor changes.

The Unison software architecture is shown in Figure 3. The microkernel or nanokernel-based approach separates and modularizes the operating system. The Unison OS modularity, easily understood architecture and POSIX compatibility give Unison much of its power to quickly solve Linux performance and support problems.

By adopting the Unison OS for Alvios Intelligent Eyewear, eSight achieved its performance goals while eliminating a major source of project cost, delays and ongoing development effort. The Unison philosophy of providing releases for the complete Unison OS once every 18 months, and upgrades to components as required, eliminated the support burden for eSight and lead to a much faster and lower cost development.

By switching to the Unison OS and the image, communications, control and display drivers that Unison provides, the performance of the system was substantially enhanced while reducing development costs.

• Image processing latency was reduced by more than 50%.
• File system performance was doubled.
• Interrupt response improved by up to 20X.
• Communications throughput doubled depending upon system setup.
• Boot time was reduced from about a minute to a few seconds.

The performance of Alvios Intelligent Eyewear improved significantly, while boot time dropped to the point of insignificance. Development time and expense were dramatically improved because of reduced driver development, software maintenance and time spent managing the “open source” aspects of the product design.

RoweBots Research Inc.
Kitchener, ON, Canada.
(519) 208-0189.
[www.rowebots.com].

eSight
Ottawa, ON, Canada.
(613) 271-9535.
[www.eSightCorp.com].

How to develop better products, save money, meet FDA/CDRH requirements more efficiently, and avoid recalls, lawsuits and nasty criminal complaints that can put you in jail

Introduction:

As a matter of full disclosure—and at the risk of appearing anachronistic—I am a veteran of the old FDA medical device wars and have the scars (and bank account) to prove it. Looking at today’s efforts of medical device vendors to “capture” market share using their current products—most of which are inappropriate to the needs of the medical device marketplace—it provides for a sense of, as Yogi Berra once put it,  “Déjà vu all over again.”

During the approximate period of 1967 to 1986 (gosh that really dates me), I brought many products to market having achieved FDA notification of compliance under Section 510k of the Medical Devices Act of 1976 (since amended in 1990 and 1997). In those days the FDA, now called the CDRH (Center for Devices and Radiologic Health), had a scant 11 weeks to respond to a 510k application or the application was automatically accorded. (Today the 510k process can take up to 2 years.) Perhaps this is why I had so many frustrating—if not ridiculous—encounters with the FDA’s Medical Device Center.

Citing just one of more than a dozen examples, I brought to market a pediatric anesthesia mask that had a detachable pacifier. We were also able to provide a fragrance that kids loved, and it made the task of the anesthesiologist a lot easier as kids picked out their mask fragrance before entering the hospital (Tutti-Frutti Bubble Gum was the overwhelming favorite). We provided documentation that showed that the pacifier held down the kid’s tongue thereby avoiding airway obstruction during the introduction of anesthesia. In addition we showed that having the patient suckle on the pacifier (when it was detached from the mask following anesthesia) provided enhanced post operative oxygenation.

The examiner ruled the combination of the mask and pacifier a “potentially lethal weapon”! This was, of course, ridiculous. We finally got the 510k compliance notification but it took 4 months.

Truth be told, I have not filed a 510k in more than 20 years and I trust that the CDRH now hires only the best from the industry. So what is it that this old-timer might offer the developers of medical devices, the executives that might run a felony risk if their company delivers a product that kills someone, as well as the embedded technology vendors that wish to increase their market share and ameliorate the coming tsunami resulting from significant decreases in discretionary DoD funding?

Risk at the Top – How Pending Legislation Might Put Your CEO in Jail

On July 31, 2008 a Senate Bill cosponsored by Senators Edward Kennedy (D – MA) and Chuck Grassley (R- IA) was filed that would require senior officers or directors of drug and medical device companies to certify under penalty of perjury that all information submitted for a product’s approval is accurate and in compliance with federal regulations.

The Drug and Medical Device Accountability Act Bill expired at the end of the two-year Senate session on December 31, 2008, but was refiled in the Senate in 2009. This is an important piece of legislation, and medical device executives should get their house in order to accommodate the provisions.

The bill provided that product applications later found to have contained false or misleading information would be subject to stiff fines (up to $5,000,000), assessed both to companies and their senior officers, who, in addition, could face jail sentences of up to 20 years. These are serious issues. Currently the CDRH has a forensic group that looks at device software only after a device has been recalled.

If enacted, the “2009 Drug and Medical Device Accountability Act” will change the medical devices industry similarly to how the Sarbanes-Oxley bill impacted corporate accountability. Laws being what they are, we should expect overkill from its enactment. This is why medical device companies’ senior management should take time to rethink their strategic approach to the delivery of their products. The bill has apparently been put on hold since Kennedy’s death, but it might be resurrected.

The FDA’s Center for Devices and Radiological Health (CDRH) reports that in 2006, 21% of all medical device recalls were for software defects—it is also estimated that one in three software-based products is recalled. They haven’t updated this data since, but one can assume that it might have gotten worse.

What’s a Medical Device Company to Do?

Strategically, the best approach that a medical device company can take is to utilize the best technology that the industry has to offer—an approach that includes:

• Requirements Definition and Management
• Change and Configuration Management
• Quality Management/Testing—Software Verification and Validation Tools
• Modeling
• Release Management
• Documentation
• Team Collaboration

Following this strategy, the ability to develop and implement a software development internal audit path to provide assurance and confidence in the integrity of the software is clearly a “best practice.” For example, using UML code reuse modeling technology, the developed software can be retained and used for future upgrades. Also, this audit trail can be used in the event of a product failure to show the CDRH forensic group that all due care was taken in the product’s development and deployment.

Do I Really Need a DO 178B Certified OS?

Don’t get caught up with industry hype for highly certified RTOSs (e.g., DO 178B Level A certification). It is a good design practice to use an OS that is specifically suited for your application. ThreadX, Micrium and Nucleus (as well as MontaVista Linux) are examples of OSs that are currently deployed in hundreds of millions of applications. Remember that for patient monitoring applications the expected critical frequency is 100 Hz—so the use of a costly OS that guarantees a 10 microsecond response is unreasonable, expensive and power hungry. Base the selection of the OS and the processor on the systems requirements. If you need secure communications, you can run your existing applications on a MILS-certified OS as a guest OS and not have to worry about rewriting existing code.

What Guidance Has the CDRH Provided That Would Help Us?

On May 11, 2005 the CDRH issued a non-binding “Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices.” Given the potential consequences of improper filings contained in the Drug and Device Accountability Act of 2009, it would constitute a best practice for any CEO to embrace the guidance provided by the CDRH.

Although the following is not required as part of the 510k filing, it makes good sense to follow these guidelines as an internal “Best Practice.”

1)    Risk assessment and management is a very important part of your filing with the CDRH and can be your best approach for protecting you under the Act. Be exceedingly careful in documenting the “Level of Concern” section of your application. It would be wise to include systems failure documentation that is available through modeling or formal methods for certification. Make sure that you specify the correct level of concern and document your approach to rectifying potential failures.

2)    Develop a Device Hazard Analysis for all software devices—include all hazards (hardware and software) associated with the product’s intended use. This can include the user GUI and how it might be operated by personnel that work under stressful conditions.

3)    Develop a Software Requirements Specification (SRS) that includes functional, performance, interface and developmental requirements for the software, including hardware, OS and programming language requirements.

4)    Develop an Architecture Design Chart (flowchart or similar illustration) that describes the relationships among the major functional units in the software device. There should be sufficient information to allow for the organization of the software relative to the functionality and intended use of the software device.

5)    The software design specification should present information to demonstrate that the work performed by the software development engineers was clear and unambiguous, with minimum ad hoc design decisions. Use a requirements management tool such as DOORS—rather than depend on Word.

6)    In a perfect world, you should be able to relate requirements with code. It’s a good idea to show the relationship between requirements, code and testing. Doing this visually (through modeling) makes the most sense as it is easier to visualize the code executing.

7)    Develop a summary of the Life Cycle plan and the Life Cycle processes employed. It will be useful to develop an annotated list of the control/baseline documents generated during the software development process, and a list or description of software coding standards.

8)    Develop verification and validation documentation and base it on the claimed Level of Concern. Whenever software is changed, a validation analysis should be conducted to validate the specific change and also to determine the extent to which this change may impact the entire system’s operation. Documentation and tracking is essential.

9)    Include a revision level history of software revisions generated during the course of product development.

Following these guidelines will minimize product failures due to software malfunction and provide an audit trail for the CDRH forensics group in the event of a product failure, to show that all due diligence was followed.

About the Author:

Jerry Krasner, Ph.D., MBA is the founder of Embedded Market Forecasters and its parent company, American Technology International (1991). A recognized authority with over 30 years of embedded industry experience, Dr. Krasner has extensive clinical research and medical industrial experience, including the successful filing of more than a dozen 510k submissions.

Dr. Krasner served as President of Biocybernetics, Inc. and CLINCO, Inc., Executive Vice President of Plasmedics, Inc. and Clinical Development Corporation, and Director of Medical Sciences for the Carnegie-Mellon Institute of Research. He has been the principal investigator of several NIH funded clinical research programs.

Dr. Krasner was formerly Chairman of Biomedical Engineering at Boston University, and Chairman of Electrical and Computer Engineering at Wentworth Institute of Technology.

Dr. Krasner earned BSEE and MSEE degrees from Washington University, a Ph.D. in Medical Physiology / Biophysics from Boston University and an MBA from Nichols College.

His papers can be seen at www.embeddedforecast.com and on his Blog www.embeddedmarketintelligence.com.

Chris Griffith, Medical Business Development Manager of Texas Instruments

Factors Driving the Growth

The driving force for the growth comes from the need to reduce healthcare costs, provide greater accessibility to healthcare services, and a desire for greater convenience.

• The healthcare industry is trying to keep healthcare costs from getting totally out of control. Healthcare costs are predicted to double the current $2.5 trillion in the next few years. This is not a sustainable model.
• Better accessibility to services is much needed by people in both industrial and developing countries. Technology can help solve “the distance problem” and make healthcare services available to more people, as well as assist with overall disease management to achieve better health for underserved populations. The idea is simple—fewer doctors can serve more people with higher efficiency using medical devices that connect patients remotely to the caretakers.
• The idea of living better and more independently while staying in one’s own home (including homecare) is being promoted by many organizations including the Continua Alliance, a leading organization that released a specification on end-to-end connectivity and interoperability (Figure 3) in health delivery systems. Additionally, the Alliance has developed a certification program to help medical devices comply with the specification. While treating patients using various medical devices is a big market, another growing area of opportunity is fitness and preventative health. This includes exercise gear and personal health monitoring gadgets. This market is aimed at consumers who want to stay fit, and the expected volume for devices is large.

Figure 1: Worldwide Medical Semiconductor Shipment Forecast. Source: Databeans Estimates.

Applications

Continua has outlined how various devices would connect to Telehealth services (caretakers). A variety of devices are already available to monitor the condition of a person who wants to stay healthy and live independently. They include digital thermometers, pulse oximeters, pulse/blood pressure monitors, weight scales, glucose meters, cardio exercise machines, electrocardiogram devices and insulin pumps. Additionally, there are medical devices used in clinical applications such as ultrasound and scanning devices, digital stethoscopes, MRI and digital X-ray. How have semiconductors shaped the design landscape? Over the years, the features of lower power combined with more functions, including the front-end input/output (I/O) into a single chip, have made medical devices more portable. See photos of portable ECG device (Figures 4) from Philips Healthcare and portable ultrasound scanners (Figure 5) from GE Healthcare. Homecare devices, such as blood pressure monitors and glucose meters, are frequently battery operated. Overall they are more compact and convenient to use.

Figure 2: Worldwide Medical Semiconductor Market Forecast. Source: Databeans Estimates.

Wireless technologies are becoming more and more commonplace with many new medical devices starting to integrate wireless features into the application. The Omnipod insulin pump developed by Insulet consists of two units: a pump worn by the patient with built-in wireless capability, and a handheld controller with built-in glucose meter. The user can control the delivery of insulin wirelessly. Omnipod provides great convenience to the userasitcanbeworn24hoursadayandisespecially desired by people with active life styles, including many athletes. To accomplish this, Insulet has used a microcontroller and wireless chip from Freescale, as shown in Figure 6. With the advancement of semiconductor technology, there are plenty of solutions available from many suppliers for medical device designers to choose from. What should a designer expect from a semiconductor supplier? Better support. Let us illustrate this with a thermometer design. Texas Instruments offers a single chip solution that includes a lowpower, single-chip AF4110 microcontroller with built-in LCD driver (Figure 7). It comes with a reference design circuit schematic, printed circuit board (PCB) layout and the bill of materials (BOM). The designer only needs to follow the design and make some custom adjustments to deliver a complete digital thermometer with an accuracy of +/0.1 degree centigrade and a reading range of 31 to 43 degree centigrade.

Figure 3: Continua’s vision of personal medical devices.

Overall, semiconductor suppliers have done a good job in integrating the front-end analog-to-digital (A/D) functions in a single piece of silicon. (For a detailed list of product offerings from various semiconductor suppliers, go to www.medsmag.com/sbb). Leading suppliers offer different solutions. Texas Instruments, Freescale and STMicro have the broadest portfolio including digital thermometers, weight scales, ECG/EKD/EEG electrocardiograms, glucose meters, insulin pump, pulse oximeters, blood pressure monitors and ultrasound/scanning devices. Separately, ADI and STMicro offer a MEMS motion detect solution for fall-detection and prevention devices. The innovation of ECG development is moving from portable to wearable. This solution will directly reduce healthcare costs. By wearing an ECG device, a patient with a heart problem can be monitored remotely by the caretaker without being in the hospital. STMicro’s battery-powered ECG semiconductor will be a good fit. More and more devices are connected to other devices/ controllers remotely using emerging wireless standards such as ANT+, Bluetooth, ZigBee and near field communication (NFC). Companies like Renesas, TI and Freescale all offer products in these areas under the umbrella of Mobile Health (commonly known as mHealth), or Wireless Health.   Another important segment in medical electronic device design is that of sensors. Most people know wide area network (WAN) or local area network (LAN). Now a new term called BAN is emerging. It is the body area network in which the body acts as a network to connect to a medical device. It works by having a sensor connected to the human body and communicating electronic signals to the receiving device much like electrodes are connected to a human body. The sensor can be a passive device (does not require power) or an active device (requires power). A new innovation from STMicro can energize an active sensor without using a battery. The M24LR16ER product is based on RFID technology, which receives power from a remote controller sending RF signals to the sensor.

Figure 4: Portable Pagewriter TC70 Cardiograph ECG Device from Philips Healthcare.

Figure 5: Portable Vscan Ultrasound Device from GE Healthcare.

Figure 6: Insulet Omnipod insulin pump with wireless control.

While all the companies above focus on many homecare devices, Intel is taking a different approach by offering point-of-care stations and hospital bedside entertainment systems based on the Atom processor. These are embedded devices with new applications. The Intel-based Point-of-care system from Kontron is such an application.

Intel-based Point-of-care system from Kontron

Another vision Intel has is to enable developers to build high-end fitness machines where a PC-like display is mounted on a treadmill that would communicate with sensors or devices worn by the users. This provides feedback to the users while they are running on the machine. Additionally, the high-end graphics display can provide personal entertainment making exercising more fun.

Figure 7: Texas Instruments thermometer (front and back): Complete reference design of single-chip digital thermometer from TI.

Design Choice

So among all these semiconductor suppliers, is there a clear winner? Choosing a chip to design a medical device is a complex process. According to design consulting firms Sterling Smartware and LogicPD, who specialize in medical product design, features and power requirements are the key factors in selecting semiconductors for medical devices. Additionally, life cycle management and supply chain management are important, as medical products do not change as fast as consumer products and require longterm vendor support. BCS Innovations, another design consulting company with offices in the U.S. and Australia, further suggested that the design cycle should cover component selection, design process and production support (with ISO 13485 certification) to be able to yield high-quality medical devices. “As users demand portability and more compact design, component counts and production processes such as package-on-package should be part of the design process,” suggested David Bull, CEO of BCS Innovations. (Package-on-package is a manufacturing process in which two chips are stacked together like a high-rise building to reduce space.)

Semiconductor solutions by segments

Semiconductors have enabled medical electronic device reductions in size, cost and power consumption while boasting significant increases in overall performance. “Thanks to the semiconductor development, consumers can now buy pulse oximeters, blood pressure meters, blood glucose meters and bathroom scales for well under $50,” according to Chris Griffith, Medical Business Development Manager of Texas Instruments. “And further integration of more functions on a single chip is expected in the future.” “There will be integration of wireless, configurable analog and embedded processing power on-chip enabling designers more flexibility in design,” commented Steve Dean, Global Healthcare Segment Lead of Freescale.

Steve Dean, Global Healthcare Segment Lead of Freescale

Expect to see device developers continue to race to bring new products to market putting more functions into more compact designs. The demand for medical electronic devices will continue to be strong not only in the U.S. but in many other regions such as Asia and Europe. The ongoing challenges will be designing products that are easy to use by non-technical users, and that these devices will be able to connect to each other and share data securely and reliably with ease. This is easier said than done.

I have witnessed Mobile Health in action. Many ECG devices are portable, but now I have started to see wearable ECG. Last year I saw one by Imec. This year at CES I saw another wearable ECG using clothing as a sensor. It is made by Cardiosport based in the UK. (I like the company name).

John Koon, Publisher

The implantable defibrillator is in a class all by itself. A friend of mine has an implantable defibrillator, and I have learned a lot from him about how his life depends on it. I came to appreciate the human aspect of it. The battery has limited life. Changing the battery means a hospital visit, where the doctor has to cut the body open to replace the battery or the unit. Average battery life is 5 years. I have heard new batteries with a 10-year battery life are being worked on. This means an additional 5 years without surgery. What about the concept of charging the unit wirelessly much like charging a cell phone wirelessly by placing it in a cradle? I imagine the user would only need to lie in a magnetic field to get charged up after a night’s sleep. But then I don’t know enough about how that magnetic field would affect the body.

Even though I am talking here about how the ECG devices are changing, the pattern of change applies to all the other medical electronic devices as well. This is partly due to the magic of semiconductors, which pack a lot of functions into a very small package. (See this issue’s “Overview of Medical Semiconductor Market and Applications.”) However, the brain of all these medical electronic devices is, in fact, in the software. It is a very important part of medical electronic device design. We will investigate this further in our next issue.

Tom WIlliams, Editor-in-Chief

There is, however, one small disturbing item in the context of the contest, and that is the use of the word “diagnose.” The contest stipulates that, “the winner will be a device that can most ac- curately diagnose a set of 15 diseases across 30 consumers in three days.” It further states that these diagnoses will leverage technology innovation in areas such as artificial intelligence and wireless sensing to make medical diagnoses independent of a physician or healthcare provider. While I’m not a lawyer, it appears that this could be getting into questionable legal territory—like practicing medi- cine without a license. Of course, just saying, “My tricorder says you have beri beri,” is not going to get anyone in trouble, but acting on that judgment without the input of a physician just might. After all, even in the Star Trek series the tricorder was always wielded by Dr. McCoy, who was a medical doctor—as in, “Dammit Jim, I’m a doctor, not a bricklayer!”

The ever-growing number of today’s medical electronic devices give us data. They do not con- tain the large amount of artificial intelligence gleaned from four years of medical school, internship, residency and experience, to shape that data into a reliable diagnosis for much of anything beyond the sniffles. There is some further hubris in the contest material that talks of, “transforming health- care by turning the ‘art’ of medicine into a science.” Now, really. Anyone who has seen even a few episodes of “House” should be able to appreciate how subtly difficult it can be to arrive at a reliable diagnosis even with vast amounts of data and test results. I can easily imagine MDs taking offense at such a suggestion.

Interestingly, though it appears that the Competition Guidelines will soon state the full details, there does not presently appear to be a list of exactly which diseases are to be diagnosed. It does say, “This diagnosis must be performed in the hands of a consumer independently of a healthcare worker or facility.” That’s where things could potentially get dicey.

I think it is important that we understand just what these devices are supposed to be. They are extensions of the physician’s knowledge, skills and art—not substitutes for it. They can be extremely valuable in a world where those skills are at a premium and where we can delegate large amounts of routine data gathering and, yes, some analysis to machines. But until we arrive at the stage where we have actual medical knowledge and experience in the form of a virtual doctor on our holodeck, as in the later series, “Star Trek Voyager,” let us please keep perspective.

I am excited to see what comes out of this ambitious competition. I am certain the effort will result in some very impressive advances, and we will be looking forward to reporting on them in these pages. But if we get too arrogant about our devices, we may face Spock looking down his nose and shaking his head at our obsession with our “beads and rattles.”

Charles B. Sidebottom, P.E., Director, Corporate Standards, Medtronic Inc.; Secretary, IEC SC 62A; Secretary, ISO TC 150, SC 5

A Dangerous Dilemma

Alarm fatigue is not just a nuisance; it can contribute to patient harm. In recent years, the U.S. Food and Drug Administration (FDA) has received hundreds of reports of patient deaths that are traceable to alarm-system- related issues. The ECRI Institute now ranks alarm-system-related hazardous situations as number 1 on their Top 10 list of Health Technology Hazards. The Joint Commission and the FDA have announced they are working on developing a systematic strategy to address alarm fatigue.

In support of these efforts, the Association for the Advancement of Medical Instrumentation (AAMI), in partnership with the FDA, the Joint Commission, the American College of Clinical Engineering, and the ECRI Institute, held a multidisciplinary stakeholder Alarm Summit in October 2011. Clinicians, manufacturers, biomedical professionals, researchers, acoustic experts, regulators, and patient safety advocates came together to share their perspectives on the challenges and opportunities surrounding clinical alarms, and to identify priorities that must be addressed to improve patient care. Among them is strengthening medical electrical equipment standards and contracting language to promote success in all intended use environments.

Responsive Standards

In 2003, IEC Subcommittee (SC) 62A, Common aspects of electrical equipment used in medical practice, published the first comprehensive safety standard that addresses medical alarm systems, IEC 60601-1, Medical electrical equipment – Part 1: General requirements for basic safety and essential performance. The standard deals with a wide array of challenges, including:

• Standardizing a vocabulary to describe alarm states and conditions
• Developing a prioritization of alarm signals based on urgency of action
• Harmonizing alarm signal inactivation states and their indications
• Providing for consistent use of color and rhythm to indicate alarm condition priority
• Permitting intelligent (smart) alarm systems and distributed alarm systems

Originally, some considered the alarm system collateral standard to be optional. Subclause 1.3 of IEC 60601-1:2005 resolved that confusion by declaring that “applicable collateral standards become normative at the date of their publication and shall apply together with this standard.” In addition, IEC 60601-1-8 is recognized by the FDA and is a harmonized standard under the European Medical Device Directive.

The IEC SC 62A Alarm Systems Joint Working Group, which is co-convened by John Hedley-Whyte, M.D., with Harvard University and David Osborn, just completed work on an amendment to the second edition of IEC 60601-1-8, which is scheduled for publication later this year. This amendment deals with several key issues including certain testing requirements, a clarification of alarm conditions priorities, and the introduction of a new “alarm acknowledged” state that has been requested by clinicians and manufacturers.
The Alarm Systems Joint Working Group is now laying plans for work on a third edition of the standard to address some of the issues raised at the Alarm Summit. These include:

• Strategies for escalating priorities so true priorities won’t be left dangerously unattended over an extended period of time
• Additional requirements for smart alarm systems that use multiple signal inputs from the patient to assess priority
• More comprehensive requirements for distributed alarm systems, as many alarm signals will soon be delivered to wireless devices held by clinical operators, including in the home healthcare environment

Thinking outside the Box

Another issue that emerged at the 2011 Alarm Summit was how few healthcare facilities are tailoring the alarm systems in today’s equipment. It may be that the clinical operators don’t have too few options, but too many. Anyone who has worked extensively with Microsoft® Office can understand the problem. One quickly realizes that there are hundreds and hundreds of configuration options available – so many that only the “super user” has the time and the inclination to understand how they work together. So in most cases the system gets used as it comes out of the box.

The same is true with many medical alarm systems. They get used as configured by the manufacturer “out of the box,” regardless of whether the equipment is being used in the operating room, intensive care unit, cardiac care unit, or the general patient environment. Given these very different environments of use, is it any wonder that clinical operators are overwhelmed with alarm signals that often turn out to be nuisances?
In 2007, IEC/SC 62A, in partnership with International Organization for Standardization (ISO) Technical Committee (TC) 210, Quality management and corresponding general aspects for medical devices, published a comprehensive standard dealing with the usability of medical devices. IEC 62366 Medical devices – Application of usability engineering to medical devices, specifies a process for a manufacturer to analyze, specify, design, verify, and validate usability as it relates to the safety of a medical device. Since 2010, compliance with IEC 62366 is required for conformity with IEC 60601-1.
When applying the usability engineering process to the alarm system, the manufacturer needs to focus on the relationship between the operator and the technology – and, above all, the manufacturer needs to keep it simple.

Manufacturers also have to recognize that the clinical operator is often dealing with multiple devices from different manufacturers at the same time. Using common terminology and easily configurable patient-relevant alarm limits and providing common alarm system indications and controls are steps forward. The current standards address many of these issues, but more needs to be done. The next generation of alarm systems standards, already in the planning stage, will address a number of the use issues raised at the Alarm Summit.

A Group Effort

The standards mentioned above are intended for the manufacturers of medical equipment. However, the Alarm Summit clearly pointed out that alarm fatigue can only
be adequately dealt with by the combined efforts of all the stakeholders. To address the safe integration of alarm systems as well as other healthcare technology systems,
IEC SC 62A, in partnership with ISO TC 215, Health informatics, has developed IEC 80001-1, Application of risk management for IT networks incorporating medical devices – Part 1: Roles, responsibilities and activities. Several technical reports in the IEC 80001 series that address risk management in a highly networked environment are due for publication in 2012. Also, at the request of the Alarm Systems Joint Working Group, work has begun on a new technical report on alarm system integration that focuses on safety, effectiveness, and data and system security. A draft report is planned for release in 2012.

At the Alarm Summit, Frank Block, M.D., co-chair of AAMI’s Alarm Standards Committee and a member of the Alarm Systems Joint Working Group, challenged the standards community. Dr. Block asserts that “alarms – and alarm standards – need to be designed as a system, and not just as a ‘box.’” He identified several areas where the standards development process and the product could be improved. These include:

• Increased clinician participation and input
• Application of knowledge on the design of medical alarm systems in other, well studied fields, such as manufacturing processes, nuclear power plants, and aviation and air traffic control
• Specificity on acceptable response time for “immediate” response to high-priority alarm signals or “prompt” response to medical-priority alarm signals
• Attention to whether devices should sound alarm signals as well as to their priority and urgency
• More information on how to create or use intelligent, integrated, unified, or distributed alarm systems, which are mentioned in the standards

The Alarm Systems Joint Working Group intends to work on several of these issues in the next edition of IEC 60601-1-8. In the near term, the AAMI Alarms Committee will focus on the issue of alarm system management with new standards, technical information reports, and guidance documents for industry and users.
In her opening remarks to the Alarm Summit, AAMI president Mary Logan laid out a vision that by 2017 no patient will be harmed by an adverse alarm event. While today’s medical alarm system may be out of control, there is much that can be done both in the short and long term to improve safety. To achieve this goal in just five short years will require the concerted and collaborative effort of all the stakeholders. We in the international standards community are committed to doing our part.

Renee J. Michalkiewicz, Trace Laboratories, Inc.

Please read attached article for specific details on the lucrative business of component counterfeiting and how you can avoid being a victim.

Overview

A worldwide epidemic of counterfeit electronic components is flooding the market and affects the supply chains of all industries.  It is estimated that the financial loss due to counterfeit components is well over $10 billion per year.  According to Thomas Hallin, an intellectual property attorney at Greensfelder, Hemker & Gale, P.C., in Chicago and former chief litigation counsel in the IP (Intellectual Property) Practice Group at Ford Motor Company,  “The multi-billion-dollar counterfeit industry, particularly in China, is costing the U.S. auto industry billions of dollars in annual sales and precluding the employment of hundreds of thousands of workers because of lost business.”  Hallin also added  that the counterfeit parts problem also raises important safety issues.¹  With regard to high end electronics specifically, “the Alliance for Gray Market and Counterfeit Abatement (AGMA) estimates one out of every 10 IT (Information Technology) products are counterfeit or contain partial counterfeit parts.”²  Counterfeiting itself becomes profitable when scrapped components, components from recycled products, or inexpensive components can be “remarked” and sold as a new, more expensive, higher reliability version.  Much of the effort today has not been placed on preventing counterfeiting but rather screening components to identify and remove counterfeits before they are used in a finished product.

As with any counterfeiting, be it money, designer clothing, or electronic components, there is a battle between the counterfeiter and the industry affected.  Each tries to better their ability to either fool or recognize the other.  Counterfeit components entered the marketplace and the electronics industry countered by adapting a variety of existing test methods to help screen components for authenticity.  These methods have proven effective in detecting fakes before they enter the product stream and have become the conventional techniques used in the war on counterfeiting.   They are becoming more and more familiar to engineers and purchasing agents and are often added to purchasing documents to insure the authenticity of incoming supplies.  Unfortunately, these techniques and their limitations are also becoming more familiar to the counterfeiters themselves.  With this knowledge, counterfeiters are able to improve their craft and utilize materials and processes that can allow a fake component to evade detection.

Because counterfeiting is so lucrative, counterfeiters are motivated to keep improving the techniques that will allow them to stay in business.  The onus has now fallen back on the electronics industry to improve its techniques to detect this “next generation” of counterfeit components.  In addition to the use of conventional screening techniques, a variety of unconventional techniques is being explored to stay ahead of the counterfeiters.

Double marking evident on part on left

Reasons for Proliferation of Counterfeiting

The motivation behind counterfeiting electronic components is the same as any other counterfeiting operation – profitability.  There are millions of dollars to be made with, currently, little risk to the criminal. The origins of these counterfeit parts are now well known and they truly represent a situation in which we are reaping what we have sown.  The U.S. was aware that electronic waste contained a multitude of hazardous substances but remained unwilling to restrict the use of these substances, deciding instead that it would be advantageous to sell and export our waste for disposal in poorer countries, who were more concerned with money than pollution.  However, before this waste made it to the landfill, it passed through the hands of entrepreneurs who removed anything they could potentially use.  The used and potentially inoperable electronic components that these individuals removed were refurbished and/or relabeled and resold back to the U.S. as new parts.  Today’s counterfeiting operations have grown from a simple cottage industry to complex operations run by organized crime that produce highly realistic-looking parts.

Findings Based Upon US Department of Commerce Report

In a report issued in January of 2010, the Office of Technology Evaluation (OTE) summarizes the state of US counterfeit electronics concerns.

• “all elements of the supply chain have been directly impacted by counterfeit electronics;
• there is a lack of dialogue between all organizations in the U.S. supply chain;
• companies and organizations assume that others in the supply chain are testing parts;
• lack of traceability in the supply chain is commonplace;
• there is an insufficient chain of accountability within organizations;
• recordkeeping on counterfeit incidents by organizations is very limited;
• most organizations do not know who to contact in the U.S. Government regarding
• counterfeit parts;
• stricter testing protocols and quality control practices for inventories are required; and
• most DOD (Department of Defense) organizations do not have policies in place to prevent counterfeit parts from infiltrating their supply chain.”³

Texture differences evident between top and bottom due to “blacktopping”

So why does it seems that so little is done to deter counterfeiting?

Well, a variety of reasons act together in preventing an organized attack against counterfeiting.  First, many counterfeits, particularly those that operate like the original, though typically not of the same quality, often go undetected and are installed into the finished product.  When a counterfeit is suspected, it is frequently difficult to confirm as the inspectors typically do not know all the subtleties of the authentic part.  Compounding the problem, Original Component Manufacturers (OCMs) are often unwilling to aid in the identification of suspect parts purchased outside of their approved distributors.  They, rightfully, want to sell current products or products through approved sources and do not want to encourage the use of unauthorized vendors.

Second, even if a counterfeit is detected, there is not one central clearinghouse for this information.  Thus, when a counterfeit is detected, companies typically just refuse to pay for them and discard them.  There are several organizations, such as ERAI, that compile counterfeit information but the sources are only their member companies.  Thus, there are likely far more counterfeits being detected than being reported throughout the industry.

Third, there is a stigma associated with possessing counterfeits.  Companies which originally reported that they had discovered counterfeit parts on incoming inspection were quickly criticized by media outlets, and associated with counterfeit components.  A tarnished reputation was immediately felt by the mere association with counterfeit parts even though these companies may have been more diligent than their competitors in preventing counterfeit parts from entering their finished product.  A fear of reporting counterfeit detection developed, and if the crime is not reported, there is little that can be done to prevent it.

Fourth, the law enforcement and government agencies involved in counterfeit prevention have limited resources.  There are numerous organizations that have agents and individuals investigating and developing plans to deal with counterfeit electronic components; the FBI, ICE, IRS, Defense Criminal Investigative Service (DCIS), Naval Criminal Investigative Service (NCIS), DOD, NASA, Government Accountability Office (GAO), and many others are all aware of the problem.  However, in regard to the main investigative agencies, the FBI and ICE, the electronic community does not lobby for action as the apparel, jewelry, pharmaceutical, music, and film industries do.  Virtually all of the investigative resources go towards industries other than electronics.⁴  This may soon change.  “The Senate on Tuesday approved an amendment by the bipartisan leadership of the Senate Armed Services Committee to strengthen protections against a flood of counterfeit electronic parts coming into the defense supply system.  Sens. Carl Levin, D-Mich., and John McCain, R-Ariz., the chairman and ranking member of the committee, offered the legislation as an amendment to the National Defense Authorization Act for Fiscal Year 2012. The legislation is a response to a committee investigation that found more than 1,800 instances of counterfeit electronic parts in the defense supply chain. It now becomes part of the authorization act, which is being debated on the Senate floor.”⁵

All these reasons conspire against a concerted effort to prevent counterfeiting and keep the exact monetary losses unknown.  So, instead of focusing on prevention, the companies within the electronics industry currently, individually, focus on finding and eliminating counterfeits on a case-by-case basis. This is costly and inefficient.  Thus, the need for screening techniques developed. 

Mitigating the Risk

So can you afford to keep the blinders on?  What are the costs?

• Costs to replace failed parts
• Lost sales
• Lost brand value or damage to business image⁶
• Safety concerns

Lead Frame and die shape differences on components submitted as single lot

As reported by the US Department of Commerce, the following steps should be taken to minimize the risk of counterfeit component infiltration into the electronics market:

• “provide clear, written guidance to personnel on part procurement, testing, and inventory management;
• implement procedures for detecting and reporting suspect electronic components;

• purchase parts directly from OCMs and/or their authorized suppliers when possible, or require part traceability when purchasing from independent distributors and brokers;
• establish a list of trusted suppliers – which can include OCMs, authorized suppliers, independent distributors, and brokers – to enable informed procurement and develop an untrusted supplier list to document questionable sources;
• utilize third-party escrow services to hold payment during part testing;
• adopt realistic schedules for procuring electronic components;
• modify contract requirements with suppliers to require improved notices of termination of the manufacture of electronic components and of final life-time part purchase opportunities;’ensure physical destruction of all defective, damaged, and substandard parts;
• expand use of authentication technologies by part manufacturers and/or their distributors;
• screen and test parts to assure authenticity prior to placing components in inventory, including returns and buy backs;
• strengthen part testing protocols to conform to the latest industry standards;
• verify the integrity of test results provided by contract testing houses;
• perform site audits of supplier parts inventory and quality processes where practical;
• maintain an internal database of suspected and confirmed counterfeit parts; and report all suspect and confirmed counterfeit components to federal authorities and industry associations.”⁷

Trace Laboratories is available to help you begin a counterfeit detection program.  The program can be as simple or as complex as you require; you determine the acceptable risk.

References

1. 1.     Labuzinski, Randy, Counterfeit Parts – Especially From China – Threaten U.S. Auto Industry’s Recovery, Jaffe Legal News Service, August 8, 2011,  (Direct Link: http://www.jlns.com/top-stories/2011/08/08/counterfeit-parts-%E2%80%93-especially-china-%E2%80%93-threaten-us-auto-industry%E2%80%99/31725).
2. New Momentum, Inc., Fighting High Technology Counterfeiting with High Technology Solutions,  (Direct Link: http://www.bizforum.org/whitepapers/newmomentum.htm).
3. U.S. Department of Commerce, Bureau of Industry and Security, Office of Technology Evaluation, Defense Industrial Base Assessment: Counterfeit Electronics, January 2010. (Direct Link: http://www.bis.doc.gov/defenseindustrialbaseprograms/osies/defmarketresearchrpts/final_counterfeit_electronics_report.pdf).
4. Radman, John and Philips, Daniel, Novel Approaches for the Detection of Counterfeit Electronic Components, InCompliance Magazine, October 2010, (Direct Link: http://www.incompliancemag.com/index.php?option=com_content&view=article&id=461:novel-approaches-for-the-detection-of-counterfeit-electronic-components&catid=26:design&Itemid=130).
5. Targeted News Service Report on News Release issued by the office of Sen. Carl Levin, D-Mich, (Direct Link: http://www.militaryaerospace.com/index/display/wire-news-display/1552319871.html)
6. Aerospace Industries Association, Counterfeit Parts: Increasing Awareness and Developing Countermeasures, March 2011, (Direct Link: http://www.aia-aerospace.org/assets/counterfeit-web11.pdf).
7. U.S. Department of Commerce, Bureau of Industry and Security, Office of Technology Evaluation, Defense Industrial Base Assessment: Counterfeit Electronics, January 2010. (Direct Link: http://www.bis.doc.gov/defenseindustrialbaseprograms/osies/defmarketresearchrpts/final_counterfeit_electronics_report.pdf).

Renee Michalkiewicz Bio

Renee Michalkiewicz, MS, MT (ASCP) is the General Manager of Trace Laboratories in Baltimore, MD.  She has been with Trace for 17 years. Trace (www.tracelabs.com) was established more than 30 years ago and offers chemical, electrical, environmental, mechanical, and advanced analytical testing services.

rmichalkiewicz@tracelabs.com / 410-229-4360

Of the many models for healthcare evolution, connected healthcare (sometimes termed the digital hospital), is emerging as an effective solution to some of the challenges of an aging population, a deficit of medical professionals and the rising cost of healthcare. With connected healthcare, providers seamlessly share clinical information stored in vast databases rather than maintained on single-instance paper charts, which helps improve care delivery and quality along with patient safety.

More intelligent and better connected medical devices can improve the data driven into the Electronic Medical Record (EMR), which is the cornerstone of connected healthcare. But, as the complexity and the number of devices within a clinical environment increase, so do the challenges associated with integrating, managing and securing them.

There may be as many as 20,000 different pieces of technology and applications throughout a hospital and even more across a healthcare network. This dynamic and wide range of implemented technology creates a massive interoperability issue. There is little or no interoperability across multiple functions and applications across a hospital network environment.

Intel Active Management Technology (Intel AMT), one element of Intel vPro technology, is built into select Intel processors and chipsets and offers an avenue to address those interoperability issues with a framework for management and security. It provides mechanisms for remote discovery, repair and protection of computing systems to improve the efficiency of remote management and asset inventory solutions by providing persistent connectivity, either wired or wireless, that doesn’t require the computing system to be functional.

Traditionally, remote management consoles communicated with devices using their standard networking capability, called the “in-band” link. The drawback to this approach is that the majority of a device has to be functional (e.g., operating system, hard drive, CPU and network drivers). In contrast, Intel AMT circuitry establishes a new communications channel, called the “out-of-band” link that operates independently of the computing system and enables communication with, and control over, non-functioning systems.

Intel AMT enables the remote discovery of medical devices in any operational state. It stores hardware asset information in flash memory that can be read anytime, even if the device is currently shut down. Intel AMT also enables the management console to diagnose, control and repair devices after software or operating system failures. System security software is remotely updated with the most recent patches. The presence and operation of cyber-protection can be confirmed and monitored centrally.

A working Intel AMT-enabled system consists of the Intel-based hardware, a management console and a provisioning system. The hardware can be a standard laptop, workstation, or server whose motherboard is based on any of the Intel chipsets that support Intel AMT technology. Increasingly, these motherboards are found embedded in medical devices such as intelligent nursing carts, pharmaceutical dispensing machines and medical tablets.

The management engine (ME) is the brains of the AMT system and is made up of a computing core located in the system’s platform control hub (PCH). The management engine runs from the motherboard’s 3.3V standby power. As such, the management engine is active even when the system has been shut down. The management engine doesn’t depend on a running host operating system to perform its functions.

The firmware that runs in the ME is known as the management engine BIOS extension (MEBx). The MEBx firmware contains a full TCP/IP network stack, drivers for the Intel AMT network interface(s), security protocols for both access and traffic such as 802.1X, TLS and SOAP/HTTPS, a full graphical keyboard-video-mouse (KVM) server, network filters and other Intel AMT applications discussed below.

For a system to be Intel AMT-enabled, it has to be built using select Ethernet and/or WiFi interfaces whose drivers are included in the ME firmware. Intel AMT-enabled motherboards include a certain amount of NVRAM that is allocated for storage of the ME firmware image, hardware and software asset information, and storage of network security keys. This NVRAM is typically shared with the overall system for the storage of the BIOS image and any optional ROMs needed for the network interfaces. The Intel AMT-enabled system reserves a portion of the system RAM for execution space much like the BIOS does.

Management Console
For systems to be managed via Intel AMT, some sort of external management console is needed. This management console runs one of many Intel AMT-aware systems management applications such as Symantec’s Altiris, Microsoft’s SCCM, or LANDesk. These management software packages are typically already used in a hospital IT environment. Additionally, an Intel AMT high-level API (HLAPI) is available that allows custom Intel AMT management applications to be built, and Intel provides fully functional reference management applications that allow for Intel AMT management for smaller businesses, technology trials and sample HLAPI code that helps in the development of customized Intel AMT management applications. In the following examples, the Intel Manageability Reference Console will used to illustrate many of the features of remote out-of-band management using Intel AMT.

Provisioning System
When a new Intel AMT-enabled system is introduced into the hospital’s IT network, that Intel AMT-enabled system must go through a one-time provisioning for it to be accessible to the management console. Intel AMT provisioning can be accomplished in a number of ways depending on the size of the network and the security policies enforced by the hospital’s IT organization. Many of the commercial management applications listed previously have Intel AMT provisioning functions built into them. Additionally, Intel provides software tools for Intel AMT provisioning that can be run on a central provisioning server or run directly on the system to be provisioned (host-based provisioning). Provisioning can also be run from a specially set up USB memory drive, or via direct entry into the Intel AMT system’s MEBx setup screens (much like BIOS setup).

Intel AMT Features
Even though Intel AMT enables outof-band management, it can still leverage host management agents to enhance management operations when the host OS is available and running properly (for instance graceful shut down).

Remote Power Management, as shown in Figure 1, is a feature of Intel AMT that allows the Intel AMT-aware management console to request a graceful shut down of the host operating system, force a shut down of a non-responsive operating system such as one that has hung or “blue-screened,” or request a system that is currently shut down to power up either to the BIOS setup screen or the operating system.

With systems based on Intel AMT versions 7.0 and greater, a fully graphical remote KVM was added to the ME firmware as shown in Figure 2. This allows the management console to run a KVM client such as RealVNC, UltraVNC, or pcAnywhere and attach to that ME firmware-resident KVM server. The IT technician gets a fully graphical view of the video console of the remote medical device and can start applications using the keyboard and mouse just as if he were at the device.

Similar in intent to the KVM feature, the Serial-over-LAN feature (SOL) allows for the redirection of the remote device’s serial console to the administrator’s console. This is useful for medical devices that don’t have a graphical interface but a simpler serial interface for control and monitoring.

The IDE Redirection capability (IDER) within Intel AMT allows the IT administrator to redirect the remote device’s CDROM or floppy to a CDROM or ISO image on the administrator’s workstation. Once this IDE-R session is established, the administrator is able to boot the remote device using an ISO image on his local CDROM or disk. This is frequently used to boot a failed device with a diagnostic image or recovery image, as shown in Figure 3, and even allows the administrator to remotely reformat and reimage a device. This feature is usually used in conjunction with either the KVM or SOL feature.

With Intel AMT, all network traffic to and from the host OS passes through the Intel AMT firmware’s watchful eye. This is accomplished using configurable Intel AMT filters that can block packets coming from or destined to the host OS. It also allows for specific Intel AMT management packets to be sent directly to Intel AMT firmware instead of going to the host OS. In normal operation, Intel AMT would allow all non-Intel AMT packets to flow unhindered between the host OS and the Ethernet or WiFi interface. However, if the host OS becomes corrupted with malware that starts spewing dangerous packets onto the network, the administrator can sever the network connection to the OS but still retain out-of-band network control. This action can be taken manually by the administrator or automatically based on network threat detection policies configured into the management console. The administrator can then deal with the infected system remotely and thus prevent the potential spread of malware or denial-of-service attacks from threatening other systems on the hospital’s network.

Intel AMT also includes hardware and software inventory data that is stored in the system’s NVRAM, as shown in Figure 4. This information can be queried anytime whether the machine is up or down. The management console can have policies configured to check this inventory information on a periodic basis to detect changes in things like amount of installed memory and raise alerts if that occurs. There is also an area in NVRAM for the system’s OEM to add OEM-specific inventory information such as device name, model number, or serial number.
Intel AMT includes very strong security features for both access security and network traffic security. Most of the major network security protocols are supported. This guarantees that management traffic between the device and the management console can be both strongly authenticated and strongly encrypted. It also allows for the network traffic to be similarly secured using the SOAP/HTTPS protocol, providing the ironclad network security needed for HIPAA/HITECH compliance.

Emerson Network Power, Intel and Symantec have developed a working proof of concept platform as an example of how the electronic medical record and Intel vPro can address connected healthcare (Figure 5). The proof of concept has been implemented as both an automated medication dispensing system called MedDispense and a wireless mobile workstation. At the heart of this proof of concept is an Emerson Network Power embedded motherboard based on the latest Intel Core i7 processor, supporting Intel AMT. The addition of Symantec’s Altiris Solution provides a large part of the management functionality.
Emerson Network Power

Tempe, AZ. (602) 438-5720.