Get ahead of stringent design cycles by evaluating the human factors and usability of a device early in the process. When the demands for meeting these requirements come from the FDA, as they almost certainly will, your product and your team will be ready.
by David Hirning MS and Virginia A Lang, PhD, HirnLan, Inc.
Whether at the blackjack table or in the marketplace, is it not preferable to have the deck stacked in your favor? Wouldn’t you rather play blackjack knowing you have a significant advantage of winning? The same is true with developing new products. With development and production costs continuously escalating and profit margins shrinking, anything that can be done to minimize development costs and maximize market preference will improve product profitability.
When the FDA started requiring human factors/usability testing of medical devices, they were actually stacking the deck in favor of medical device manufacturers. Yes, that is right. Human factors/usability testing is actually a win/win for medical device manufacturers. Let me explain.
Human Factors Engineering is based on research, scientific/experimental method, statistics, human physiology and cognitive information processing. It is the combination of engineering, human physiology, behavioral performance and cognitive science. Human Factors Engineering is a scientific discipline that studies how humans interact with devices, products and/or systems. It approaches design with the user as the focal point. Human Factors Engineering ensures that devices, products and systems are safe, effective and usable by their intended users.
The FDA requires device manufacturers to create a use-based risk analysis for their products. By creating this risk analysis, medical device manufacturers do a thorough examination of their product(s) from the point of view of the user. The risk analysis then becomes the basis of the human factors/usability testing. One important aspect of the risk analysis is the proposed mitigations for high-risk/high-frequency tasks. Instructions for use, training and product patient inserts are no longer considered appropriate mitigations. Instead, appropriate mitigations must be related to the design of the device. So, how do you know that these design mitigations are accurate? That is where human factors/usability testing comes into the picture.
Here is where you can start stacking the deck in your favor. You see, Human Factors have methods to evaluate those mitigations early in the product development cycle. The FDA calls these formative testing/methods. These tests are evaluative in purpose. Getting your product in front of users will quickly tell you if your design mitigations are accurate. In addition, it will give you the opportunity to make changes to your product very early in your product development cycle. As you know, identifying product problems early in the cycle will enable lower cost changes than if you waited until just before launch. What goes into a formative test?
Formative testing uses a prototype of the device and puts it into the hands of users.
Formative testing sits at the center of the development cycle to align the actual model of human interaction with the ultimate design of the device or system.
But how do you identify the users of your products? The first thing is that users and customers are not always the same. Users are those who will personally use your product. There could be more than one user of your product. You need to identify all the users and how they would use your product. You are identifying these users for the purposes of testing your product, not for marketing. You are identifying these users so that you can put your product into their hands early in the product development cycle.
How many users do you need for the formative testing? The magic number is eight, yes eight—eight from each user group that would have the highest risk/highest frequency tasks using your product. For example, if the device were a ventilator, then there are several different types of users: the technician who calibrates the ventilator, the respiratory therapist, the nurse, the caregiver and the physician. Do you need eight of each for the formative testing? No. You need the users who perform the highest risk/highest frequency tasks. In this case it would be the technician, the respiratory therapist and the caregiver, for a total of 24 test participants. Why the caregiver? The FDA is paying particular attention to caregivers who use medical devices, even if it’s just to monitor the output. The reason the FDA considers caregivers to be the highest risk user group is because they are not professionally trained.
Your next questions are probably: how long does formative testing take, and how much does it cost? In terms of time, two weeks of preparation, two to four days of testing and two weeks to obtain the report. Yes, that amounts to about five or six weeks. Now, the cost—that depends upon a number of different variables. If your user groups are medical professionals, then it is going to cost more for participant recruiting and incentives than if your user groups are caregivers or patients. There are costs for the facilities used for the testing, creation of the test plan, facilitation of the test and the report writing. I know you would like a number. Formative tests can cost as little as $25,000 up to about $55,000. These are only rough estimates.
The FDA recommends formative human factors/usability testing, but does not require it. So why should you do it? Remember, you are trying to stack the deck in your favor for getting a product that is safe, effective, usable and approved by the FDA in the shortest time possible. Finding out you have some serious design problems with your device early in the development cycle will cost less to fix than if you find out at the end of the product development cycle. You might actually want to do two or three rounds of formative human factors/usability testing. Then when you get to the summative testing, you are assured that you have mitigated all of the risks through design changes.
What is a summative human factors/usability test? The summative test is the final test, which renders a pass/fail judgment on a device, product or system. This test validates that the device, product or system is safe, effective and usable by the all intended user groups. It differs from the formative test in that now you have to use a device that represents exactly the device that is going to be launched to the market. So you are not using prototypes. In addition, you must have at least 15 users for each and every user group that will use your product. In the example of the ventilator, that means you would have five different user groups, or a minimum of 75 test participants. Each user group in this case would have different tasks with respect to using the ventilator and so the test tasks will be different for each group. I think you can figure out that the cost of a summative human factors/usability test is considerably more than a formative human factors/usability test. I am sorry to say that providing an estimated cost is very difficult. With that said, in our experience we have found that the minimum cost is about $50,000. That is for one user group comprised of non-medical professionals.
Here are a few questions I am frequently asked:
Question 1: If formative tests are not required, why should I spend the money?
Answer: Formative tests provide answers in the early stages of product development, when changes are relatively small compared to changes required after a summative test.
Question 2: Why should I conduct a formative test on my information for use?
Answer: In much the same way a formative test looks at a prototype providing design insight early for design mitigation, the information for use evaluation study (a formative test) can mean the difference of passing/failing a summative test. We have seen summative test disasters directly related to the information for use.
Question 3: Marketing has already put the product in front of users and the users all liked it and would purchase it. The users even said they thought it would be safe and usable. Why can’t I just use those results?
Answer: Marketing uses product concepts and asks users to give their opinion. Human factors/usability testing is evaluating user performance using the product in an environment that is similar to where the product would be used. These are one-on-one testing sessions using strict scientific/experimental methods.
Question 4: I understand everything, you are right and I will incorporate Human Factors Engineering into my next product. However, I am at the end of the product cycle. My product is ready to ship—what do I do now?
Answer: It’s never too late to involve a human factors engineer. You should have a human factors engineer perform an expert review to fully understand the risks moving into a summative test. You should have an Information for Use evaluation study (a formative test) completed to be sure your Information For Use does not hinder the user from successful completion of the summative test tasks. The next step, as appropriate, is to complete the summative test. Just remember, you may not pass the summative test without product design modifications.
Question 5: You keep mentioning a human factors engineer. How do I know if I am engaging a reputable firm/individual?
Answer: That’s a tough one. It would be simple to say look for an individual with a PhD in Human Factors Engineering. But remember, Human Factors Engineering is the intersection of engineering, human physiology, behavioral and cognitive science. I would suggest a graduate degree in any of those fields with experience. I would look to recommendations from friends and/or colleagues. I would be careful if the team running your testing is not the same as the team you talked to in the beginning. If it’s too good to be true—it probably is.
Question 6: I didn’t budget for human factors/usability testing. Can I submit my 510(k) to the FDA anyway?
Answer: Yes, you can submit the 510(k). However, if your device is considered a Class 3 or Class 2 device, is a combination device, an infusion pump, a ventilator, a surgical instrument, a monitoring device, or a device used by patients and or/caregivers, then you will get “the letter.” The letter will state that you need to do human factors/usability testing.
So imagine a blackjack table that allows you to turn over all the cards and actually arrange the cards before they are dealt. This is what the FDA has provided with the human factors requirements. With each formative test you get to arrange the cards. You will know early in the development cycle what the risks are and will have mitigated them—when the cost of change is still minimal. Run an evaluation study on the Information for Use (a formative test). You will then know that the Information for Use is accurate, understandable and easy to use.
It’s now time to run the summative test. How much did it cost to engineer/design your product? Don’t worry, let it ride. The cards are dealt…Blackjack! But you knew that, you stacked the deck. You submit your 510(k) to the FDA…Blackjack again! But again, you knew that, you stacked the deck. You followed best practice during the product design/development process.
The recommendations and requirements of the FDA to do human factors/usability testing are actually the same best practice for creating safe, effective, usable and profitable products in any industry. Minimize development costs, maximize market preference—the result, profitable products. Sound familiar? How many iPhones are used everyday? Embrace best practice, build safe, effective and usable products. Stack the deck. The FDA’s recommendations and requirements empower you to stack the deck in your favor.
Medical Device Human Factors by HirLan, Carlsbad, CA. (619) 301-2073. [www.MedicalDeviceHumanFactors.com].
