I received a bunch of emails from people regarding my previous post on PCORI and I was also quoted in a recent Politico article on PCORI punting on looking at diseases and instead spending more money and hiring consultants to yet again decide what it should hire more consultants and hold more meetings about what it should study. I received a form letter from PCORI's executive director Joe Selby that reads in part:
An initial research agenda with the goal of deciding what broad questions a neither research nor starting point, it's group therapy.
In my previous post I wondered by the National Pharmaceutical Council would hold a conference on asymmetries in information coming out of PCORI and made a somewhat snarky comment about not knowing what asymmetry is with respect to PCORI unless it mean dividing up it's grant money to stakeholders.
It was explained to me that 'asymmetry' has to do with an extremely important issue: Namely while that PCORI, AHRQ, etc will not only determine what information is sent out to doctors, patients, health plans but will also control the means of production and distribution. Unlike NIH, what PCORI will put out can and will be used in determining what health plans should and will pay for. But given PCORI and AHRQs financial focuse on dissemination it begs the question of whether the mostly outdated information that will be distributed via government web sites, conferences and academic detailers will crowd out other forms of information and studies.
The NPC event on asymmetry will be important if it raises these questions http://www.npcnow.org/Public/Education___Events/Events/2012_events/Asymmetry_2012/asym12.aspx
Further, PCORI is loaded with what are called a heuristic biases. I encourage anyone who doesn't think that PCORI is designed to establish asymmetries to read Judgment under Uncertainty: Heuristics and Biases Amos Tversky; Daniel Kahneman Science, New Series, Vol. 185, No. 4157. (Sep. 27, 1974), pp. 1124-1131. http://links.jstor.org/sici?sici=0036-8075%2819740927%293%3A185%3A4157%3C1124%3AJUUHAB%3E2.0.CO%3B2-M
Tversky and Kahneman describe how subjective assessment of probabliities leads to systematic and severe errors in judgment. Even more so if the subjective assessement is also systematic.
Why? As T and K explain: judgments are all based on data of limited validity, which are processed according to heuristic rules.
What are these heuristics? The first one of representativeness is used to assess a person, product, or event based on how similar it seems to be of stereotypes or preconceptions we already have in mind. PCORI uses representativeness by assessing, funding and disseminatiing CER in terms of how well it fits the subjective belief that all CER will lead to lower costs and better health. That is, if a particular approach to research and dissemination fits the pre-existing stereotype of CER then PCORI assumes it will improve health delivery. Moreover, any objectiion raised to CER by introducing information basedon the prior probability, or base-rate frequency, of the outcomes of CER is ignored. Simply stated, PCORI never looks at the actual relevance and impact of CER to determine if it effective. Rather, the problem is one of poor dissemination, not design or scholarship. This insensitivity to prior outcomes is a hallmark of representative bias and explains the total lack of PCORI's self criticism.
Second, PCORI and AHRQ institutionalize insensitivity to sample size. If a study compares two products and is representative of the desired CER outcome then it is accepted. It adheres to the law of small numbers," according to which even small samples are highly representative of the populations from
which they are drawn. PCORI's is framed by the expectation that a valid hypothesis about a population will be represented by a statistically significant result in a sample- with little regard for its size. So for instance Steve Pearson and Sean Tunis, two people who's organizations stand to be enriched by PCORI cash cite clinical trials that 'concluded' injection of medical cement into compression fractures of the spine produced no better pain relief than "sham" injections and the FDA's withdrawing approval of Avastin for advanced breast cancer (claiming there were no benefits given the risks) as examples of good CER.
But both ignore the extremely small sample sizes of studies for both issues. Rather, the small studies are not only highly representative of the population as a whole but representative of what good CER is. And this exposes another, more profound bias institutionalizd by PCORI, name that of availability. We remember and regard as true based on what information happens most often or is more available. Hence, because PCORI and AHRQ will produce and deliver most of the information on medical treatments and will control the outlets for such information it uses the heuristic of availability to create the perception that the CER it produces is the "right" path. Or as Kahneman quips: "What we see is what it is."
Finally, PCORI reinforces the view that only CER and evidence based medicine it produces should be talked about, shared and acted upon. So what if consumers with their own genetic information, companies that produce novel results, or physicians want to learn more about individualize care want to share and use such knowledge. It turns out that they can't: Under the health care law all the actionable information for benefit coverage and reimbursement is channelled through AHRQ and PCORI. It's ok for AHRQ funded academic detailers to wine and dine docs about the blessings of CER but it is illegal for innovators to talk about and share scientific knowledge about the novel uses of their products. In otherwords some medical knowledge (PCORI/AHRQ produced CER generated by the people who have a subjective bias that CER is wonderful) is more equal than others.
There is more to discuss regarding asymmetry and there will be more to come. But as you can see, I believe I was wrong to dismiss asymmetry as a Beltway slogan. Asymmetry is a real problem. As Tversy and Kahneman note: "The rational judge will nevertheless strive for compatibility,
even though internal consistency is more easily achieved and assessed. In particular, he will attempt to make his probability judgments compatible with his knowledge about the subject matter, the laws of probability, and his own judgmental heuristics and biases."
The way PCORI Is structured and the way in which AHRQ generates and produces CER, incentivizes the use of these biases. By seeking to generate acceptance of CER and embedding results into clinical decision tools, PCORI and AHRQ makes "internal consistency" of the research it's funds the foundation of medical decsionmaking. It thereby institutionalizes several errors in judgment that will leave patients less healthy.
|
|
|
In my previous post I wondered by the National Pharmaceutical Council would hold a conference on asymmetries in information coming out of PCORI and made a somewhat snarky comment about not knowing what asymmetry is with respect to PCORI unless it mean dividing up it's grant money to stakeholders.
It was explained to me that 'asymmetry' has to do with an extremely important issue: Namely while that PCORI, AHRQ, etc will not only determine what information is sent out to doctors, patients, health plans but will also control the means of production and distribution. Unlike NIH, what PCORI will put out can and will be used in determining what health plans should and will pay for. But given PCORI and AHRQs financial focuse on dissemination it begs the question of whether the mostly outdated information that will be distributed via government web sites, conferences and academic detailers will crowd out other forms of information and studies.
The NPC event on asymmetry will be important if it raises these questions http://www.npcnow.org/Public/Education___Events/Events/2012_events/Asymmetry_2012/asym12.aspx
Further, PCORI is loaded with what are called a heuristic biases. I encourage anyone who doesn't think that PCORI is designed to establish asymmetries to read Judgment under Uncertainty: Heuristics and Biases Amos Tversky; Daniel Kahneman Science, New Series, Vol. 185, No. 4157. (Sep. 27, 1974), pp. 1124-1131. http://links.jstor.org/sici?sici=0036-8075%2819740927%293%3A185%3A4157%3C1124%3AJUUHAB%3E2.0.CO%3B2-M
Tversky and Kahneman describe how subjective assessment of probabliities leads to systematic and severe errors in judgment. Even more so if the subjective assessement is also systematic.
Why? As T and K explain: judgments are all based on data of limited validity, which are processed according to heuristic rules.
What are these heuristics? The first one of representativeness is used to assess a person, product, or event based on how similar it seems to be of stereotypes or preconceptions we already have in mind. PCORI uses representativeness by assessing, funding and disseminatiing CER in terms of how well it fits the subjective belief that all CER will lead to lower costs and better health. That is, if a particular approach to research and dissemination fits the pre-existing stereotype of CER then PCORI assumes it will improve health delivery. Moreover, any objectiion raised to CER by introducing information basedon the prior probability, or base-rate frequency, of the outcomes of CER is ignored. Simply stated, PCORI never looks at the actual relevance and impact of CER to determine if it effective. Rather, the problem is one of poor dissemination, not design or scholarship. This insensitivity to prior outcomes is a hallmark of representative bias and explains the total lack of PCORI's self criticism.
Second, PCORI and AHRQ institutionalize insensitivity to sample size. If a study compares two products and is representative of the desired CER outcome then it is accepted. It adheres to the law of small numbers," according to which even small samples are highly representative of the populations from
which they are drawn. PCORI's is framed by the expectation that a valid hypothesis about a population will be represented by a statistically significant result in a sample- with little regard for its size. So for instance Steve Pearson and Sean Tunis, two people who's organizations stand to be enriched by PCORI cash cite clinical trials that 'concluded' injection of medical cement into compression fractures of the spine produced no better pain relief than "sham" injections and the FDA's withdrawing approval of Avastin for advanced breast cancer (claiming there were no benefits given the risks) as examples of good CER.
But both ignore the extremely small sample sizes of studies for both issues. Rather, the small studies are not only highly representative of the population as a whole but representative of what good CER is. And this exposes another, more profound bias institutionalizd by PCORI, name that of availability. We remember and regard as true based on what information happens most often or is more available. Hence, because PCORI and AHRQ will produce and deliver most of the information on medical treatments and will control the outlets for such information it uses the heuristic of availability to create the perception that the CER it produces is the "right" path. Or as Kahneman quips: "What we see is what it is."
Finally, PCORI reinforces the view that only CER and evidence based medicine it produces should be talked about, shared and acted upon. So what if consumers with their own genetic information, companies that produce novel results, or physicians want to learn more about individualize care want to share and use such knowledge. It turns out that they can't: Under the health care law all the actionable information for benefit coverage and reimbursement is channelled through AHRQ and PCORI. It's ok for AHRQ funded academic detailers to wine and dine docs about the blessings of CER but it is illegal for innovators to talk about and share scientific knowledge about the novel uses of their products. In otherwords some medical knowledge (PCORI/AHRQ produced CER generated by the people who have a subjective bias that CER is wonderful) is more equal than others.
There is more to discuss regarding asymmetry and there will be more to come. But as you can see, I believe I was wrong to dismiss asymmetry as a Beltway slogan. Asymmetry is a real problem. As Tversy and Kahneman note: "The rational judge will nevertheless strive for compatibility,
even though internal consistency is more easily achieved and assessed. In particular, he will attempt to make his probability judgments compatible with his knowledge about the subject matter, the laws of probability, and his own judgmental heuristics and biases."
The way PCORI Is structured and the way in which AHRQ generates and produces CER, incentivizes the use of these biases. By seeking to generate acceptance of CER and embedding results into clinical decision tools, PCORI and AHRQ makes "internal consistency" of the research it's funds the foundation of medical decsionmaking. It thereby institutionalizes several errors in judgment that will leave patients less healthy.
