Nirit Weiss, MD (a neurosurgeon with a Master of Business Administration and a specialization in healthcare economics) has written a blistering appraisal of heavily reported and oft cited July 2008 Commonwealth Fund report, “Why Not the Best?”
After reading her analysis, perhaps a better title would have been “Why Not the Best Use of Statistics?”
In addition to entirely discrediting the report’s methodologies, the article (in a series of very disturbing sidebars) shows that the mainstream media was either lazy or complicit (or both) in reporting the story.
Weiss’ full exposé can be found here.
Weiss writes:
“On July 17, 2008, the Commonwealth Fund commission published its second Scorecard report, Why Not the Best?: Results from the national scorecard on U.S. Health System Performance, 2008. Its findings that U.S. health care performance had not improved since 2006 and that access to health care significantly declined were again reported widely in the media and in a way that gave wide credence to the Scorecard’s conclusions (see sidebar). As the New York Times noted, The findings are likely to provide supporting evidence for the political notion that the nation’s health care system needs to be fixed.”
However, the 2008 Scorecard must be interpreted with caution. In attempting to diagnose the ills of America’s healthcare system, the Scorecard suffers from serious flaws that challenge the validity of its conclusions – flaws that were, essentially, ignored by the authors of the study and completely missed by the media coverage. These flaws fall into three categories:
1. The methodology by which the data were collected and the studies were designed to address specific questions.
2. Arbitrary definitions and metrics used to define the concept of “quality” in health care.
3. Sweeping, broad conclusions that are unsubstantiated by the findings of the study.
1. Flawed methodology
The 2008 Scorecard is based upon multiple disparate studies, using various methodologies, non-uniform definitions of “benchmark,” and arbitrary assumptions as to what “logical policy goals” are, and what “adequately insured” actually means. The Scorecard attempts to draw meaningful conclusions based on a summation of individual studies with varying sample sizes, varying performance comparisons, and varying data collection techniques.
In peer-reviewed, scientific literature, it is invalid to lump together the results of multiple studies, using multiple methodologies, in the same charts, graphs, and conclusions, without assigning relative weight to the results of the studies. Adding even more to the confusion, many of the reported data are not directly referenced to published studies, so it is impossible to trace and evaluate the sources of the information. A substantial number of the individual analyses were merely described as “conducted by the authors,” limiting the reader’s ability to evaluate the quality and validity of the studies.
Perhaps more troubling are those data which can be traced back to their source studies, and turn out to be based on patient self-reporting. For example, in its section on “quality,” the Scorecard quantified mistakes made in health care delivery according to patient interpretation and self-reporting! Patients were asked how often they felt mistakes were made in their medical care, such as errors in laboratory testing, or medication errors.
2. Arbitrary metrics used to define “quality”
The entire premise of the Scorecard, is that “quality” in the U.S. Healthcare System can be quantified by assessing mean scores with respect to 37 “indicators” across five “dimensions” of health system performance. The 37 “indicators” across five “dimensions” are modeled after those used in studies of industry, and focus on health care delivery systems performance, which is indeed one component of value and return-on-investment. But it is not the largest determinant of what most Americans would define as “quality.” Assessing the quality of delivery of goods by studying uniformity, for example, is appropriate when evaluating the transformation of undifferentiated inputs into uniform outputs, each machined to be identical to the other.
In other words, the conclusions of the study are dependent on the authors’ assumption that all patients with a given diagnosis, say diabetes, are otherwise identical, and should have no difference in outcomes. This input/output calculation disregards the fact that all inputs, such as patients with diabetes, have other comorbidities, and cannot be expected to have the same outcome or outputs.
3. Unsubstantiated conclusions
The authors of the Scorecard present their results in raw form, and then apply these results to reach broad conclusions, which are several steps removed from the data actually being presented; indeed, the data presented often does not directly support or relate to some of the Scorecard’s conclusions. Because there are so many such examples, it is reasonable to summarize them. In general, the authors conclude that universal health insurance in the U.S. would improve quality at substantial cost-savings, that increased primary care and preventative care always improves outcomes at reduced cost, and that “preventable” hospitalizations and outcomes are preventable entirely by policy changes in health care delivery, in the absence of a drastic culture shift in this country. These conclusions may have some validity, but they are not directly supported by the data presented.
Given the potentially far-reaching implications of publications such as the 2008 Scorecard on policy-making and financing of healthcare, the authors and sponsors of these studies must be held to the same high standards as are physicians and scientists when reporting results of their investigations. This responsibility must be shared by the media, who control the dissemination of information, and must present the results of such studies in as objective and informative a manner as their audience deserves.
As Trevor Butterworth comments in a sidebar:
“Both the Washington Post (“U.S. Health Care Still Ill, Survey Finds”) and Forbes (same headline) ran an article by Healthday News, which simply reported the Scorecard's conclusions and cited Karen Davis, the Commonwealth Fund's president, and Cathy Schoen, the Commonwealth Fund's senior vice president for research and evaluation, to explain the significance:
“‘The United States also lags behind other countries in health-care results, Schoen said. ‘Even where the U.S. average improved, other countries have improved much more rapidly,’ she said. ‘As a result, we are falling further behind the leaders.’”
No other expert opinion was offered in either article, but Post readers were invited to visit the Commonwealth Fund's website “To learn more about health care in the United States.”
A half-truth, as my grandmother used to say, is a whole lie.
After reading her analysis, perhaps a better title would have been “Why Not the Best Use of Statistics?”
In addition to entirely discrediting the report’s methodologies, the article (in a series of very disturbing sidebars) shows that the mainstream media was either lazy or complicit (or both) in reporting the story.
Weiss’ full exposé can be found here.
Weiss writes:
“On July 17, 2008, the Commonwealth Fund commission published its second Scorecard report, Why Not the Best?: Results from the national scorecard on U.S. Health System Performance, 2008. Its findings that U.S. health care performance had not improved since 2006 and that access to health care significantly declined were again reported widely in the media and in a way that gave wide credence to the Scorecard’s conclusions (see sidebar). As the New York Times noted, The findings are likely to provide supporting evidence for the political notion that the nation’s health care system needs to be fixed.”
However, the 2008 Scorecard must be interpreted with caution. In attempting to diagnose the ills of America’s healthcare system, the Scorecard suffers from serious flaws that challenge the validity of its conclusions – flaws that were, essentially, ignored by the authors of the study and completely missed by the media coverage. These flaws fall into three categories:
1. The methodology by which the data were collected and the studies were designed to address specific questions.
2. Arbitrary definitions and metrics used to define the concept of “quality” in health care.
3. Sweeping, broad conclusions that are unsubstantiated by the findings of the study.
1. Flawed methodology
The 2008 Scorecard is based upon multiple disparate studies, using various methodologies, non-uniform definitions of “benchmark,” and arbitrary assumptions as to what “logical policy goals” are, and what “adequately insured” actually means. The Scorecard attempts to draw meaningful conclusions based on a summation of individual studies with varying sample sizes, varying performance comparisons, and varying data collection techniques.
In peer-reviewed, scientific literature, it is invalid to lump together the results of multiple studies, using multiple methodologies, in the same charts, graphs, and conclusions, without assigning relative weight to the results of the studies. Adding even more to the confusion, many of the reported data are not directly referenced to published studies, so it is impossible to trace and evaluate the sources of the information. A substantial number of the individual analyses were merely described as “conducted by the authors,” limiting the reader’s ability to evaluate the quality and validity of the studies.
Perhaps more troubling are those data which can be traced back to their source studies, and turn out to be based on patient self-reporting. For example, in its section on “quality,” the Scorecard quantified mistakes made in health care delivery according to patient interpretation and self-reporting! Patients were asked how often they felt mistakes were made in their medical care, such as errors in laboratory testing, or medication errors.
2. Arbitrary metrics used to define “quality”
The entire premise of the Scorecard, is that “quality” in the U.S. Healthcare System can be quantified by assessing mean scores with respect to 37 “indicators” across five “dimensions” of health system performance. The 37 “indicators” across five “dimensions” are modeled after those used in studies of industry, and focus on health care delivery systems performance, which is indeed one component of value and return-on-investment. But it is not the largest determinant of what most Americans would define as “quality.” Assessing the quality of delivery of goods by studying uniformity, for example, is appropriate when evaluating the transformation of undifferentiated inputs into uniform outputs, each machined to be identical to the other.
In other words, the conclusions of the study are dependent on the authors’ assumption that all patients with a given diagnosis, say diabetes, are otherwise identical, and should have no difference in outcomes. This input/output calculation disregards the fact that all inputs, such as patients with diabetes, have other comorbidities, and cannot be expected to have the same outcome or outputs.
3. Unsubstantiated conclusions
The authors of the Scorecard present their results in raw form, and then apply these results to reach broad conclusions, which are several steps removed from the data actually being presented; indeed, the data presented often does not directly support or relate to some of the Scorecard’s conclusions. Because there are so many such examples, it is reasonable to summarize them. In general, the authors conclude that universal health insurance in the U.S. would improve quality at substantial cost-savings, that increased primary care and preventative care always improves outcomes at reduced cost, and that “preventable” hospitalizations and outcomes are preventable entirely by policy changes in health care delivery, in the absence of a drastic culture shift in this country. These conclusions may have some validity, but they are not directly supported by the data presented.
Given the potentially far-reaching implications of publications such as the 2008 Scorecard on policy-making and financing of healthcare, the authors and sponsors of these studies must be held to the same high standards as are physicians and scientists when reporting results of their investigations. This responsibility must be shared by the media, who control the dissemination of information, and must present the results of such studies in as objective and informative a manner as their audience deserves.
As Trevor Butterworth comments in a sidebar:
“Both the Washington Post (“U.S. Health Care Still Ill, Survey Finds”) and Forbes (same headline) ran an article by Healthday News, which simply reported the Scorecard's conclusions and cited Karen Davis, the Commonwealth Fund's president, and Cathy Schoen, the Commonwealth Fund's senior vice president for research and evaluation, to explain the significance:
“‘The United States also lags behind other countries in health-care results, Schoen said. ‘Even where the U.S. average improved, other countries have improved much more rapidly,’ she said. ‘As a result, we are falling further behind the leaders.’”
No other expert opinion was offered in either article, but Post readers were invited to visit the Commonwealth Fund's website “To learn more about health care in the United States.”
A half-truth, as my grandmother used to say, is a whole lie.
