Unintended consequences – more on pneumonia

The current issue of the Archives of Internal Medicine has another important article on pneumonia diagnosis – Antibiotic Timing and Errors in Diagnosing Pneumonia

They also feature this editorial – Measuring the Performance of Performance Measurement

First the article -

Reduction in the required TFAD (time to first antibiotic dose) from 8 to 4 hours seems to reduce the accuracy by which ED physicians diagnose pneumonia, while failing to reduce the actual TFAD achieved for patients.

These results are no longer news, nor surprising. By emphasizing early treatment, we more likely treat on “shaky grounds.” The bigger points are made in the editorial.

There is potential for negative unintended consequences associated with virtually any performance measurement. Even the most desirable intervention may be applied to inappropriate patients to ensure good scores. Therefore, focusing only on the potential negative consequences misses the point. What is most important is ensuring that any potential negative consequences are minimized and are more than balanced by improvements in care and patient outcomes. Thus, studies like these are important in that they help define the negative consequences. Only when the negative consequences are known can there be appropriate assessment of the balance of risk and benefit of using a specific performance measure.

It is interesting to contrast the many articles and editorials attacking aspects of the CMS pneumonia performance measures with the widespread acceptance of the cardiac performance measures. Why is that? I believe that the difference lies in the quality of evidence supporting the measures. The cardiac measures are based on unassailable randomized controlled data demonstrating the importance of treatments such as aspirin and β-blockade for acute myocardial infarction and angiotensin-converting enzyme inhibitor for heart failure. Thus, the effect of possible inappropriate use in some patients pales in comparison to the benefits of increasing the use of these therapies.

Unfortunately, for pneumonia, there are limited randomized data to support many of the guideline recommendations regarding blood cultures, antibiotic timing, and antibiotic choice. Thus, when unanticipated negative consequences are reported as a result of attempts to adhere to the performance measures, it would be inappropriate to dismiss them as unimportant. We just do not have enough hard evidence about the benefits to confidently state that they outweigh the risks.

Confronted with concerns about the pneumonia measures, it would have been easy for the CMS Quality Measures Health Assessment Group to label them as anecdotal uncontrolled observations. Instead, to their credit, they have been quite willing to conduct exploratory analyses to address these concerns. In response to the concern about the high cost and low yield of blood cultures, personnel from the CMS National Pneumonia Project helped to facilitate a study of blood cultures in patients with pneumonia. When this study found that false-negative blood culture results led to an increased hospital length of stay in patients with pneumonia,8 the blood culture measure was changed to target only those patients being admitted to the intensive care unit. Similarly, the potential negative consequences of striving to provide antibiotics within 4 hours prompted a recent change to a goal of 6 hours.

What lessons can those involved in guideline creation and performance measurement take from these reports? One is that there is almost always a possibility of unintended negative consequences when providers are held accountable to performance measures. This potential is likely to increase as public reporting and pay-for-performance programs become more widespread. Therefore, it is important that performance measures be based on high-quality evidence. If this dictum is followed, it becomes much easier to accept unanticipated consequences that have only a minimal effect on patient or economic outcomes. The gradual move toward the use of outcome measures instead of process measures may help by measuring overall patient outcomes instead of encouraging the use of specific interventions. Unfortunately, even with outcome measures, there can be unintended consequences. When unintended negative consequences are reported with any type of quality improvement effort, they should be promptly investigated. Such an approach will ensure the credibility of future efforts to improve the quality of medical care.

If we must have performance measures, we should measure their impact prior to adopting them.  The 4 hour rule for pneumonia has made this point clearly.