After hearing complaints about over-administration from healthcare workers, and, in particular, that the health community was binging on checklists, Science-ish knew something was awry when a poem titled Mister Tick-A-Box arrived in my e-mail inbox from an Australian surgeon-poet-friend.
Dr. Hessom Razavi turned to writing stanzas to vent his frustrations about the checklist culture he saw emerge at his eye clinic and operating theatre in Perth. He was being reduced to an automaton by the directives of “Mister Manager, a.k.a. Executive-Director-Consultant-Supervisor-Coordinator-Liaison Officer,” he scribbled. Instead of patients, he was dealing with “the boxes, the boxes, the cold creep of the b □ o □ x □ e □ s , just waiting to be ticked.”
The eye surgeon wasn’t the first to speak of checklist burnout, and “alert fatigue” over electronic health records is already a well-documented phenomenon. This led Science-ish to wonder about the evidence behind the much-lauded surgical checklist—made famous by the Harvard surgeon and New Yorker writer Dr. Atul Gawande in his book the The Checklist Manifesto. Exactly how effective is this modern medical panacea?
Science-ish rang up Dr. Marty Makary, the Johns Hopkins surgeon who helped to develop the original surgical checklist trumpeted by Dr. Gawande and ferried around the world by the World Health Organization. When asked about checklist fatigue, Dr. Makary voiced his concern that the original surgical checklist—which includes simple elements, such as confirming the patient’s identity while she is still awake; a “time-out” before skin incision, when the medical team checks in about antibiotics and possible critical events; as well as a sponge and instrument count at the end of the operation to ensure nothing is lingering within the patient—has been a victim of its own success, bloated by risk-managers and legal departments in hospitals. “Now, people have expanded them to the point where they are too burdensome, and expanded them to the point where front-line workers don’t find them helpful.” Worse still, “If they are too burdensome, people won’t use them,” he said. “Or they will just say they are using them, document they are using them, and not use them.”
A scan of the literature revealed there have been squabbles over the true benefits of checklists. Systematic reviews note there are few high-quality studies and no randomized trials. Some say the studies we do have aren’t robust enough or didn’t run for long enough to know how the checklist will play out in the real world. (This prospective study looked at the implementation of the surgical checklist and found that despite 100 per cent completion rates, “most of the individual checkpoints are either not executed as designed or not executed at all.”) As well, we only have evidence for the checklists that have been studied; not all the off-shoots with which health-care workers like Mr. Tick-A-Box are now grappling in operating rooms all over the world.
Even the original and oft-cited New England Journal of Medicine checklist study—by Dr. Gawande and others—had some serious methodological flaws. It looked at eight hospitals around the world and demonstrated that the checklist nearly halved the number of deaths after surgery, a “too good to be true” kind of benefit in medicine. Dr. Makary explained that the results may have been exaggerated by a few of the study’s limitations. In particular, the before-after study design—where investigators collect data on patient outcomes before an intervention is introduced and after—could have biased the results. There was no control group for comparison, and participants knew they were being watched so they might have been more careful than usual. The improved outcomes might have been a result of that extra care and not the checklist, a phenomenon known as the surgical Hawthorne Effect.
As well, there was the problem of applicability of the evidence. The study conflated the results of four hospitals in the developing world, with four in the developed world. But, said Dr. Makary, “There are differences in the benefit of a checklist in the developing world, in poor countries versus wealthy countries.” Some have suggested the mortality reduction would not have been as dramatic had the research been carried out in only developed-world hospitals, where safety measures such as instrument counts are already common practice.
Still, there is good evidence that checklists improve teamwork in hospitals, which in turn, reduces surgical mortality rates. Plus, the cost is minimal, and even a modest benefit means lives will be saved. Because of this cost-benefit ratio, the checklist has been adopted all over the world, and it’s now one of a few safety measures hospitals in Canada are required to report on to their health ministries (even though there is no auditing or oversight of that self-reported information).
At Toronto General Hospital, Dr. David Urbach is replicating Dr. Gawande’s study in 109 Ontario hospitals. He expects his results will be more modest than the 50 per cent mortality reduction seen in the global study. “That means the same thing as saying a checklist prevents one of every two deaths that would have occurred after surgery,” he told Science-ish. “There aren’t many things that do that.” Even without his results, Dr. Urbach argues that we adopted the checklist as a safety measure too quickly and without robust enough evidence. He would have liked to see our health ministries put other safety measures in place that have better evidence backing them.
For his part, Dr. Makary, is satisfied with the studies that show that checklists work. What we ought to do now, he said, “is examine the implementation science of checklists, ask how we can make these more user-friendly.” This is all part of a movement to measure quality and safety in medicine and make sure hospitals publicly report metrics we know improve outcomes, which is also the focus of his new book, Unaccountable. Making sure this is done in an evidence-based fashion may be just what Mister Tick-A-Box ordered.
Science-ish is a joint project of Maclean’s, the Medical Post and the McMaster Health Forum. Julia Belluz is the associate editor at the Medical Post. Got a tip? Seen something that’s Science-ish? Message her at firstname.lastname@example.org or on Twitter @juliaoftoronto