CPOE Study Draws Strong Reaction From Medical, IT Community

More than four months after it originally was published, a March 9 Journal of the American Medical Association study that analyzed a hospital’s computerized order entry system is continuing to get strong reaction from the medical and health IT communities. The study, which linked the Eclipsys TDS electric ordering system with 22 types of medical errors, has elicited concerns that the findings, or rather the characterization of the findings, could slow CPOE adoption.

The study, conducted from 2002 to 2004 at a teaching hospital, was a qualitative and quantitative look at house staff interaction with a CPOE system that a tertiary-care teaching hospital used from 1997 to 2004. Researchers from the University of Pennsylvania School of Medicine grouped the errors into two groups: “information errors” caused by fragmented data and lack of computer and information systems integration at the hospital or “human-machine interface flaws” resulting from machine rules that conflicted with “work organization or usual behaviors.”

In a reaction letter published in the July 13 JAMA, Dr. Dan Morgenstern and Anne Keillor of ACS Healthcare Solutions called the study “disingenuous at best.” Morgenstern said he was surprised to see a study of a system that the hospital was “basically tossing out.” Although there is value in the study because it validates the importance of the medical industry taking proper steps to update older systems, he said he would have liked to have seen the study run five years ago and not now, when flawed systems are being discontinued.

Others who responded to the study also expressed concern about the specific CPOE system studied. Dr. Steven Kanig, president of CustomEHR and chair of the American Medical Association‘s e-Medicine Advisory Committee, said, “It was an important article in terms of pointing out potential sources of error in computerized physician order entry and things that need to be considered in the design of systems and how people are trained to use them.” However, he added that “our concern about this specific article was that it was reporting on an early system which is now considered to be very outdated.” Kanig — along with Dr. Sam Bierstock, vice president and chief medical officer of Healthlink, and Dr. Eugenia Marcus of Pediatric Health Care at Newton Wellesley — also wrote a reaction letter published in the July 13 JAMA.

Morgenstern said his “greatest dismay” was that the study was picked up by the mainstream media. “The fact that the headlines that most people read were, ‘Computers don’t work all that well in health care,’ … send[s] the wrong message, and that is certainly a concern,” he said.

Kanig had a similar reaction to the media coverage. “Our concern was that the conclusions of that article might be generalized in such a way that people would assume that computerized physician order entry is dangerous and is not living up to its potential,” he said, adding, “We feel that CPOE does have a significant potential to truly decrease patient error.”

Ross Koppel, the study’s lead author and a self-described “die-hard supporter of CPOE,” said reaction to the study ranged from “vitriol” to enthusiasm. Koppel said some of the study critiques he read indicate that the study in some cases was either misread or not read thoroughly.

Koppel noted that there has been a great deal of attention payed to the age of the system he was studying, but he said the system was updated just months before he studied it. He said the critique that the system they studied was a “dinosaur” has to be counterweighed by the fact that it was continually updated.

In reply to the reaction letters in JAMA, Koppel and his colleagues wrote that their article “notes CPOE’s great promise” and should be viewed as “a call to focus on the organization of work, on the need to respond to evolving technologies and on methods of monitoring those technologies.”

Koppel said the Journal of Biomedical Informatics has devoted a special section to three teams of scholars writing about the CPOE study, as well as a response by Koppel and some colleagues. The publishers of the journal have made the content available for free on their Web site.

Koppel said a point that he and his colleagues made in their paper that he feels has been overlooked is that, “The issue is not the hardware – the issue is the integration of computer systems and human processes,” Koppel said. “The real analysis, ultimately, is organizational.”

Koppel currently is working on a study that will examine a new CPOE system to see if it has addressed old problems, introduced new problems or reduced other problems. He said he and fellow researchers have collected the data, and he hopes to have the data available for analysis within weeks.

Moving forward, Morgenstern said he would like to see a reputable medical journal run a multicenter study of CPOE and electronic medical records. He would like to see research on institutions that have installed modern systems, and he would like to get data on what types of systems are effective and on error rates before and after implementation.

Dr. Don Levick of Lehigh Valley Hospital said, “I think what’s needed is quantitative analysis of the reduction in errors using these new systems compared with the introduction of new errors that may be accompanying these new CPOE systems to see if there remains a statistically significant reduction in medical errors.”

Kanig said important future studies will be prospective trials that examine a set of well-identified errors and measure error rates before and after implementation of a CPOE system in a quantitative way. “I think it’s very important that we continuously re-evaluate our systems,” he said.