Post

Good or Lucky?

One barrier to change is resistance by those who have not experienced a bad event. Most of us were trained to learn from our personal experience; quite useful in many situations, especially when scientific evidence is weak or absent. Given the vast variety of situations patients present with and the limits of scientific evidence, there are many situations in which human intuition and experience are valuable. You can often hear experienced professionals say that they made a “gut” decision; one made from instinct and intuition developed through their own experiences.

However, safer practices often are aimed at reducing rare but severe adverse outcomes. A bit of math will show that personal experience often is unreliable, and a history of good outcomes should not reassure one that his or her practices are actually safe.

For example, the incidence of wrong-site surgery is roughly 1 in 100,000 cases. How many cases would an OR team have to perform without incident before they are reasonably (95 percent) sure that their practice is at least as good as the nationwide average?

They would have had to have performed about 460,000 procedures uneventfully just to be assured that they are average; to prove performance is significantly better would take even more cases.

So numbers can help persuade, if a provider resists change saying that he has never had a problem (particularly if he believes himself to be data-driven). Ask: how many times have you done that procedure? What is an acceptable rate of error? If a target rate is 1/n, then the amount of event-free experience needed is well over 4*n (a rough approximation, but good enough—in our example above 4*n would be 400,000 when 460,000 were actually needed). Those eye-popping numbers should help people see that having no problems does not necessarily distinguish between good and lucky.

References

Agresti A, Coull BA. Approximate Is Better than “Exact” for Interval Estimation of Binomial Proportions. The American Statistician. 1998;52(2):119–126.

Kwaan MR, Studdert DM, Zinner MJ, Gawande A. Incidence, patterns, and prevention of wrong-site surgery. Archives of surgery (Chicago, Ill. : 1960). 2006;141(4):353–7; discussion 357–8.

For those interested in the math, the following could be readily calculated in a spreadsheet. If you have had no adverse events in ‘n’ cases, then:

ñ = n + 3.84
ƥ = 1.92 / ñ

Risk of adverse event is less than = ƥ + 1.95 * sqrt ( ƥ / ñ * ( 1 – ƥ ))

Note: ñ means adjusted sample size (adjusted ‘n’), ƥ means adjusted observed frequency (adjusted ‘p’), and the above calculates the 95% upper limit of the Agresti-Coull Interval