William A. Hyman
Professor Emeritus, Biomedical Engineering
Texas A&M University, w-hyman@tamu.edu
Read other articles by this author
There has been much discussion about EHRs not meeting user’s needs. This often involves complaints of cumbersome data entry and/or data extraction, even within a single product and before ever getting to information exchange. Added to this is EHR prompts (or is it advice?) from Clinical Decision Support systems and associated prompt fatigue. In such criticism the software is usually not generating actual errors (eg mixing up data and patients), it is just that it is user unfriendly or perhaps even user hostile. A common explanation for unwieldy if not maddening EHR design is that the designers didn’t know anything about the clinical environment or didn’t care, ie they wrote code that was capable of performing or allowing for a certain set of transactions, but they did not properly consider and address actual use. Presumably the code met some set of requirements that were formulated at some point, although the software world appears to be much more tolerant of evolving requirements than is the traditional hardware world. Not only are they tolerant of it, they have methodologies that celebrate it such as spiral design which basically says start here and keep adding stuff until it does enough things. It has also been argued by software product developers that regulatory schemes cannot be effectively applied to software design because the process is inherently iterative. Software version numbers that require multiple digits and letters are in some ways a measure of evolutionary design as functions are added, deleted and repaired. Here we often find the delightful word “upgrade” which can mean added features or it can mean fixing something that was never correct in the first place. By way of analogy, if your new car doesn’t start on Tuesdays and the manufacturer provides a software fix that allows it to start of Tuesdays, is this an upgrade? Perhaps even better is “rollback upgrade” which means upgrading your software by going back to a previous version. To further the car analogy, this is like taking away your new car and returning your trade-in.
Although EHRs are by fiat not medical devices and therefore not regulated by the FDA (for better or worse), the FDA’s Design Controls for medical devices does have something to offer here. In general Design Controls defines a structured methodology that moves from Planning to Design Inputs, design, formal Design Review, Design Verification. Design Validation, and Design Output which is the finished product. In particular Design Controls define as different the two words verification and validation and the associated processes In other contexts these terms are sometimes used interchangeably or in combination. Documents labeled Verification and Validation, when they exist, often just lump the two activities together without distinguishing between them.
In Design Controls verification involves testing the design against the set of requirements that were generated at the beginning of the design cycle. For medical devices these requirements are called Design Inputs and are a distillation of what someone or some group thinks the end users need. Developing Design Inputs may or may not include detailed studies and surveys of actual users. Validation comes after verification and is a test of whether the verified design actually meets actual user needs. This requires in effect a clinical trial of some level, preferably involving users (and future customers) who had no prior input into the Design Inputs or the design itself. The degree to which verification testing and validation testing are the same depends on how closely the Design Inputs actually captured what real users really need, but you can’t know this until you actually do the validation.
A catchy way to describe the difference between verification and validation, which requires careful reading, is that verification tests whether you designed the thing right while validation asks if you designed the right thing. In this regard it can be noted EHR certification is a verification procedure in that it tests whether the EHR is capable of performing certain tasks without asking whether or not it performs those tasks in a manner that real users in the real environments of use will find acceptable, let alone helpful. The same issues apply to Clinical Decision Support where verification would likely mean performance against an artificial and constrained set of patient data while validation would require performance across a broader spectrum of real patients.
I have sometimes been accused of obsessing over regulatory word distinctions which are different from how people often use and understand these words. My counter argument (and defense) is that it is important for words to be used precisely, especially when there are real differences of meaning, and when these differences are potentially important in understanding why, for example, a fully capable and certified EHR may be hated by its users.