An automated continuous integration system can provide a thorough design history through triggered builds, status reports and alerts. It can promote a team process that smoothes the complex process of managing a complex software design project.
These days a continuous integration (CI) tool is much more than “nice to have” during software development. And when it comes to the design and development of a software medical device, good continuous integration practices are even more imperative. I say this as someone who has spent excessive amounts of time wading through documents and double checking references, checking traceability, versions and making sure design outputs are properly documented in a Design History File (DHF). CI shouldn’t be thought of as nice to have, rather, it is a key process in making software medical device manufacturing possible.
A CI tool can take the sometimes feeble attempts of humans to make large amounts of documentation consistently traceable and force the computer system to do what it does best—continuously. The use of a CI tool is not simply an esoteric practice for those who are fond of its incorporation. CI is an activity that successful development teams have always attempted, but they have too often failed to utilize software tools to ease the process. Going a step further, development teams can use a CI tool to simplify steps that they may never have dreamed of before!
Continuous Integration refers to the continuous compiling and building of a project tree as well as continuous testing, releasing and quality control. This means that throughout the project, at every stage, the development team will have a build available with at least partial documentation and testing included. In general, CI builds are performed in an environment that closely matches the actual production environment of the system. A CI environment should be used to provide statistical feedback on build performance, tests and incorporation of a version control system and ticketing systems. In a development environment, the team may use a version control tool to link to tickets. In this way, any CI build will be linked to a specific change set, thereby providing linkage to issues, requirements and, ultimately, the trace matrix.
CI builds should occur frequently enough that no window of additional version control update occurs between commit and build, and such that no errors can arise without developers noticing them and correcting them immediately. This means that for a project that is in development, it should be configured that a checking triggers a build in a timely manner. Likewise, it is generally a good practice for the developer committing a change set of code to verify that his or her own change set does not break the continuous integration build. There is little overhead to creating many CI builds. There is, however, potential downside to not performing CI builds frequently enough.
Most software engineers think of a build as the output of compiling and linking. I suggest moving away from this narrow definition and expanding it. A “build” is a completion (in both the compiler sense and beyond) of all things necessary for a successful product delivery. A CI tool runs whatever scripts the development team tells it to run. As such, the team is free to use the CI tool as a build manager. It can compile code, create an installer, bundle any and all documents, create release notes, run tests and alert team members about its progress.
Jenkins CI
For the purposes of this article, the focus will be on one specific CI tool, Jenkins CI. This is one of the more popular open source tools available. Jenkins CI, the continuation of a product formerly called Hudson, allows continuous integration builds in the following ways:
1. It integrates with popular build tools (ant, maven, make) so that it can run the appropriate build scripts to compile, test and package within an environment that closely matches what the production environment will be.
2. It integrates with version control tools, including Subversion, so that different projects can be set up depending on projection location within the trunk.
3. It can be configured to trigger builds automatically by time and/or change set (i.e., if a new change set is detected in the Subversion repository for the project, a new build is triggered).
4. It reports on build status. If the build is broken, it can be configured to alert individuals by email.
Figure 1 gives an example of what a main page for Jenkins CI (or any CI tool) may look like. It can be configured to allow logins at various levels and on a per-project basis. This main page lists all the projects that are currently active, along with a status (a few details about the build) and some configuration links on the side. These links may not be available to a general user.
Clicking any project (“job”) links to further details on the build history and status. This image provides us details on what the overview screen in the CI environment might look like, but it is at the detailed project level that we see the real benefit of packaging that can be performed by a well set up CI environment.
DHF Requirements
Medical device software is audited and controlled by standards defined by the FDA, specifically Code of Federal Regulation, Title 21, parts 11 and 820 (21 CFR). Many of the requirements laid out in this difficult-to-understand guidance can be made easy when we use a CI environment throughout the course of project design and development. Looking specifically at the quality system requirements laid out by 21 CFR Part 820.30, Subpart C – Design Controls, it becomes apparent that a good CI environment can help to address each. A major consideration, perhaps the major consideration, is the completeness of the Design History File (DHF).
Figure 1 The main page of a CI tool will allow the user to navigate to various projects and levels within projects. It will also establish a system of permission levels and allow users to quickly see information on project status.
The “or reference” part of this statement stands out. Traditionally, medical device manufacturers have thought of the DHF as a physical, self-contained item. But with a project of any complexity, it isn’t difficult to imagine how quickly a DHF may grow into an unruly mess of “stuff.” Why not simply leverage software tools to make the process seamless? Using a CI tool, development teams can pull together a baseline of all the elements of a DHF as frequently as they wish to. Furthermore, they can do so with a degree of accuracy that cannot be achieved through the diligent (yet distractible) legwork of a busy team.
I propose that the DHF need not be a single physical or soft folder with duplicate copies of items. Leveraging the CI environment along with the version control system, it is a much better idea to think of the DHF as a snapshot of all relevant design outputs at a given point in time. To that end, the development team can have many snapshots of the DHF throughout the project lifecycle. To achieve this, they need simply to define this process in their standard operating procedures and work instructions. To those who have only worked with a DHF as a particular folder with specific subsets of documentation within, this approach, while it makes sense, takes a bit of a leap from the traditional DHF mindset to attempt.
The CI can present much more information than a simple build history
Figure 2 shows what a (simple) project setup may provide in the way of such packaging. It is up to the team to determine how much or how little the CI handles, but it makes the most sense to allow it to do what computers do very well and what humans tend not to do as well: align things.
The sample in Figure 2 shows that more than simply the build package can be included in the CI output. Development teams can bundle other things that are required by the DHF, including any manuals, plans, requirements and so on. This environment is starting to look more and more like a DHF.
The CI Environment
The CI build server should closely mimic the environment in which the final product will be deployed. This helps establish a level of confidence with regard to system compatibility prior to user acceptance and integration testing. It must also have access (through the version control and/or ECM system) to all the design controls and documents necessary to build a complete DHF. It is generally best to use a single version control system for everything. It doesn’t make sense, for example, to store source code in one version control system and documents in another. Doing so makes importing of all necessary items difficult, if not impossible.
There are a number of benefits to utilization of a CI server during project design and development. Do not think of CI as a tool only for software builds. Integrated with the project version control system, it can serve as much more. Within the CI build history we now have a mechanism by which we can recreate the precise environment and moment in time in which the build was generated. From code to documentation, all design history (think DHF) can be traced to the build.
The CI build gives the development team prompt feedback on the build status. If compilation fails, tests fail or some requirement element cannot be packaged, the entire team is flagged immediately. To this end, the entire team will know that a particular check-in has broken something. This feedback will eliminate the fear that an unknown break could be so extensive that progress will come to a screeching halt. The near real-time feedback of the CI build saves valuable time (and stress!) throughout development and even design.
Project progress tracking (tickets, tests, etc.) gives the team and management the ability to see a quick overview of project trends, including completion of the project, test results, code quality, reviews, documentation and so on.
Along with project progress tracking, a major feedback loop is completed. Continuous integration provides continuous feedback for all team members. Even software peer reviews can be a part of this feedback loop, with every change set and/or ticket completion triggering a mini peer review. When a team reviews smaller portions of code by paying attention to every change set, we have more manageable reviews as well as improved team understanding of each other’s work.
Triggering Builds and Keeping Them Healthy
It is important for team members to focus on keeping the CI build in a “healthy” condition. CI builds can and do break for a number of reasons. This is to be expected.
As an example, we will use Jenkins-CI to automatically perform a CI build every hour if there is a change in the repository. The system will be configured to send emails to the development team if there is a problem with the build—i.e., if a change set breaks the CI build. It is anticipated that the CI build will break from time-to-time, however, a broken build should not be left unattended.
A common cause of a broken CI build is a lack of attention to the build script. Each developer is responsible for making certain that the ant build scripts are up to date with all required changes. We cannot rely on the build scripts that are generated by an IDE. There are certainly more possible causes that could be added to the above list. It is a good idea for each developer to trigger a CI build immediately following any Subversion commit to ensure that the CI build has not been broken. If a CI build continues to be broken without being addressed, the team leader and/or project manager may revert the offending change set and re-open any related issue.
Design history storage using CI environment
Using a CI Environment to Replace the Traditional DHF
Naturally, an important part of continuous integration is having a CI build that can be checked regularly for continued build success. This is probably what is commonly thought of as the key benefit, but there is much more to be gained. Any CI environment that is worth using will allow the team to incorporate packaging of key project items with each build. This includes important documents, tests (both manual and automated test outcomes can be packaged), requirements, design specifications and build results (deployment packages, libraries, executables, installers, etc.). The important thing to note here is the fact that, used wisely, the CI environment can provide a snapshot of all project outputs at any given point in time (Figure 3). Hopefully it is becoming clear that this gives us the possibility of automated DHF creation. Not only that, but we have a much more detailed DHF throughout the life of a project and not merely at a point in time in which a particular freeze was performed.
The DHF is much more than a loosely controlled folder with a number of documents shoved in. The DHF, when properly defined, is now all of the history that goes into the design and development of the software product, and the continuous integration environment is the glue that holds it together.
Matthew T. Rupert.
Jenkins CI.
