If you can’t measure it, the saying goes, it doesn’t count. But what if you’re measuring the wrong things?
The Access to Medicines Index (www.accesstomedicineindex.org) is an attempt to measure and compare the corporate social responsibility of both innovator (20) and generics (7) companies based on a number of different (and often quixotic) indicators. It was developed by Netherland's-based non-profit Access to Medicines Foundation (ATMF) with funding from the Dutch Ministry of Foreign Affairs, the UK Department for International Development and The Bill & Melinda Gates Foundation. It was launched in 2008, and comes out every two years.
The Index analyzes the following seven technical areas across four pillars:
Pillars:
1) Commitment (30% weight)
2) Transparency (30% weight)
3) Performance (30% weight)
4) Innovation (10% weight)
Technical Areas:
1) General Access to Medicine Strategy and Governance
2) Public Policy and Advocacy
3) R&D for Index Diseases
4) Patients & Licensing
5) Equitable Pricing & Registration
6) Technology Transfer (Capability Advancement)
7) Drug Donations and Philanthropic Activities
The Index concentrates on the global list of Low and Medium Development Countries based on the UN Human Development Index and World Bank Country Income level categories. The Index has historically covered 33 diseases including: WHO Neglected Tropical Diseases; the top 10 infectious diseases based on DALYs (Disability-Adjusted Life Years) from the WHO Global Burden of Diseases for the Low Development and Medium Development Countries; and the top 10 chronic diseases based on DALYs from the WHO Global Burden of Diseases for the Low Development and Medium Development Countries. The next index will likely broaden the disease scope to additional categories including NCDs, women’s health and some cancers. They also plan to place increased weighting on companies’ actual performance (vs. their commitments).
According the Access to Medicine Foundation, the index “aims to help poor people in developing countries gain access to medicine by encouraging the pharmaceutical industry to improve its commitments and practices related to this issue.” Since it’s a comparison, the theory is that competition among companies will drive desirous “socially responsible” behaviors. That’s a noble goal, but the devil is in the details.
As Goran Tomson, professor of international health systems research at Karolinska Institute points out, the Index’s methodology cannot be reproduced, hence it cannot be considered statistically valid.
There are also troubling issues relative to the index’s metrics for success. According to the Index’s methodological designer, Afshin Mehrpouya (an assistant professor of accounting and management control at the Paris-based HEC international business school) recently said, the only current measurements are “web hits and media coverage.” That’s not very exciting, plausible, or helpful from a health policy analysis perspective.
Another metric is the opinion of patient groups. When asked why certain patient groups were chosen (they are not named in the Index), the answer was that groups were chosen based on their “credibility.” That’s code for groups who do not accept funding from the pharmaceutical industry or may share the anti-private sector bias of the party line. At minimum, it’s a dubious selection bias.
Karolinska’s Tomson also pointed out that the index’s “review committee” consisted almost entirely of “familiar faces,” thus creating an issue of normative bias.
In other words, when did you stop beating your wife?
To offer better balance and some reference, shouldn’t the Index create a parallel metric that measures the policies and political environments of low and middle-income countries to determine whether they facilitate or hinder their citizens’ access to healthcare? How about creating an index that addresses the lack of transparency in the LDC public sector and (the 800 pound gorilla in the room) corruption. These are all polite ways of saying that the design criterion stacks the deck. But, hey, doesn’t the end justify the means?
Karolinska’s Tomson put the discussion about the index—as well as the entire ICIUM enterprise—into perspective when he said the index lacked for “higher ambitions.”
“Higher ambitions” requires that the Index do more than read its own press releases and talk with its friends in NGO-Land. “Higher ambitions” requires honesty beyond one’s own cognitive mapping.
According to Index CEO and Founder Wim Leerveld, “Today all companies have teams to deliver us the requested data as they see the relevance for them.”
Maybe. Maybe not. Interviews with participating companies showed much displeasure with both the Index questionnaire, the volume and type of information being requested by the Index and the “normative bias” of the ATMi staff. In some cases, participating companies are spending hundreds of hours to collect, verify and prepare final submissions. Is it worth it?
Here are some tough questions the Indexers must ask themselves:
Q: Is the Index moving the needle? Is there evidence that this is worth the effort that companies put into it and does it justify donor funding? Have they had an impact on financial investment patterns? Web hits and press clippings don’t cut it.
Q: What work has ATMF done to validate that their metrics are in fact the right drivers to improving access? Are they selecting an agenda being pushed by activist constituents without assessing a more full-bodied picture of what is happening in the real world?
To take one glaring example, the Index operates from the assumption that the innovator pharmaceutical industry can improve access to essential medicines. But, when you examine the WHO's model Essential Drug List, very few of the 400 or so drugs deemed essential are new, or patented or were ever patented in the world's poorest countries. In category after category, from aspirin to zithromax, in almost every case and in almost every country, these medicines have always been (or have been for many years) in the public domain. That is, the medicine is fully open to legal and legitimate generic manufacture.
There are important implications for the world's poorest patients. If these patients had reliable and affordable access to these several hundred essential medicines, all available theoretically as multi-source, that is from generics companies, then global mortality and morbidity might be cut as much as 10-20% -- a huge gain for populations around the world. Strangely, the Index gives a pass to the world's largest producers of generics drugs in India and China. Those companies are not being asked to spend hundreds of hours assembling data on their contributions to medicines access. Given the potential hugely positive impact on access to medicines, any reasonable person might ask why?
Q: Is it time to reassess the Index’s bias for certain mechanisms and tools? Its fixation on the medicines patent pool, for example unfairly demoting companies that aren't in negotiations. This doesn't seem fair. The pool is only one part of a broader landscape of what's happening around access to HIV and other treatments and it's unfair to use one tool as a measure of companies' commitment when there are other things happening that are very relevant and important.
There are ideological and activist assumptions within questionnaire’s section on tech transfer question. Is the Index looking at tech transfer as a measure of access to medicines or are they promoting industrial policy? Why isn’t in-country capacity building measured? What about efforts to fight counterfeiting? If ATMi is an ideological exercise, then innovator pharmaceutical industry should question the usefulness of the proposition.
Q: How does the Index ensure each company answers the questions in the same way to allow for an apples-to-apples comparison? Is the Index able to objectively compare companies? According to one corporate participant:
“Some of the questions were just too difficult to understand and required too much interpretation. In the Pricing & Distribution section, (Q1.9) we’re asked to provide product-specific registration status. When I asked for clarification, our Index contact told us to indicate if the product is patented in the countries in which it's registered. This is not inherent in the question. Also, we think it would be useful for the Index to know where we have filed for registration but where the application may be stuck in a bottleneck since the speed of registration is highly dependent on the speed and efficiency of local countries' regulatory processes. But our contact indicated this information was not necessary.”
The concern among participating industry is that the Index is getting increasingly arbitrary and opinionated. This is exacerbated by the fact that (per ATMi staff bios on its website) – many of the Index staff has worked for NGOs such as Médecins sans Frontières and other like-minded anti-pharmaceutical industry groups.
Industry wants the Index. They value being recognized for their good work – and the competitive rankings are appreciated (and flaunted). But the agenda isn’t appreciated. Pushing TRIPS+ and penalizing intellectual property rights trivializes the nature of the program. Perhaps it’s inaccurate to say that industry wants the Index. Maybe a better statement is that industry wants an index.
Recognizing the inherent flaws and bias of the ATMi – but also the importance of measuring industry’s commitment to social responsibility and access, Business for Social Responsibility’s (http://www.bsr.org/) Healthcare Working Group has launched an effort to provide an industry-wide lens on this important issue. The Working Group’s efforts center on the group’s acknowledgement that solving this challenge is a human need, and a business priority that requires a close collaboration with other actors from industry, public and NGO sectors.
It will be interesting to see how these two programs compare – and which one survives.
