The Meaning of Meta-Analyses
by Gina Shaw
October 2008
Meta-analyses can play a role in advancing scientific knowledge for any given drug. But if poorly designed or hastily conducted, they can be damaging for an approved medicine.
In 2006, GlaxoSmithKline’s diabetes drug Avandia totaled more than US$3 billion in sales, putting it No. 2 on GSK’s sales list. But in the second quarter of 2007, sales dropped 22 percent in the wake of a meta-analysis published in the New England Journal of Medicine by Cleveland Clinic chief cardiologist Dr. Steve Nissen, which showed a 43 percent increase in heart attack risk and a 64 percent increase in heart attack death for Avandia users.
The meta-analysis, which pooled the results of 42 prior studies involving roughly 26,000 patients, was later echoed by the results of another, more highly selective study. This study—conducted by investigators at Wake Forest University School of Medicine and published in the Journal of the American Medical Association in September 2007—found the drug approximately doubled the risk of heart failure and increased heart attack risk by more than 40 percent. Avandia now carries a U.S. Food and Drug Administration (FDA) black box warning.
Such is the power of the meta-analysis, a quantitative synthesis of multiple studies focused on the same outcome, designed to garner an overall sense of the data on a given question. It doesn’t take a new randomized controlled trial to influence the sales of a particular drug; if a meta-analysis is significant enough, it can change a company’s fortunes.
Some industry watchers called the Cleveland Clinic meta-analysis “questionable” at the time, but the almost identical results of the more selective Wake Forest meta-analysis, which looked only at four long-term studies, seem to suggest that Nissen got it right. (GSK’s own meta-analysis found a 31 percent increased risk of heart attack, although the company says other data contradict that result.)
Power in Numbers
No one questions the importance of meta-analyses, especially in pooling data to help identify drug risks that are very real, but rare enough that they might not be recognized even in good-sized individual trials. But with meta-analyses having such power to change a drug’s outlook, it’s important to understand what makes a good meta-analysis and what makes a bad one.
The keys to a good meta-analysis include several elements, says Julian Higgins, Ph.D., a senior statistician at the Cambridge, U.K.-based Medical Research Council’s Biostatistics Unit and an extensive contributor to The Cochrane Collaboration.
“A meta-analysis should have clear criteria for including studies, a comprehensive search for studies, attention to the potential deficiencies of the studies being included and to the likelihood of reporting biases, and methodological/ statistical expertise in the team,” he says.
The latter is particularly important, says Dr. Elijah Dixon, assistant professor of surgery at the University of Calgary, in Calgary, Canada. In 2005, Dixon published a critical appraisal of meta-analyses in the general surgical literature in the Annals of Surgery. “We found that papers published by epidemiology groups were of the best quality; next were those published by surgeons and epidemiologists together, and both were better than those published by surgeons alone,” he says.
The reason for this is simple enough. “Methodologic experts can make sure you’re asking the right questions and collecting the right data the right way, in a reproducible manner,” he says. “Meta-analyses use quantitative techniques to combine trials, and the results benefit from having someone on the team who is comfortable with those techniques.”
The Nissen meta-analysis of Avandia studies avoided one common pitfall of meta-analyses—not digging deep enough. Many negative study results end up unpublished, so a meta-analysis that restricts its search to published literature will almost inevitably be hampered by publication bias.
“To be truly comprehensive, you need to go further and go to the top organizations in the field and do a literature search based on things such as meeting poster presentations—this could help find those negative or marginal results that may not have been offered for publication,” says Carol Etzel, Ph.D., an assistant professor in the department of epidemiology at the University of Texas M.D. Anderson Cancer Center, in Houston, Texas, USA. “Then, you may have to contact the presenters to get the data if it’s not given in the small abstract. This takes more time and more money, but if you really want the full breadth of the literature—the good and the bad—that’s what you have to do.”
Keep It Clean
Other problems found in meta-analyses, Higgins says, include the failure to allow for weaknesses in the material being analyzed and inappropriately combining studies. This “mixing apples and oranges” is a difficult pitfall, because it’s impossible to eliminate all heterogeneity among studies combined for meta-analysis. “Heterogeneity is arguably inevitable,” Higgins says.
But simply pooling the data of diverse studies yields unreliable results, as was demonstrated with a meta-analysis of circumcision and HIV transmission that appeared in the International Journal of STD and AIDS in January 1999. The meta-analysis, which found that the risk of infection was lower in uncircumcised men, combined the results of 33 diverse studies without allowing for their heterogeneity, an inappropriate approach to meta-analysis. When the data was re-analyzed in the same journal a year later, stratifying by study, the authors found that an intact foreskin was
associated with an increased risk of HIV infection: combined odds ratio 1.43 (1.32 to 1.54) with a fixed effect model and 1.67 (1.25 to 2.24) with a random effect model.
“It’s important to respect the integrity of each study design, in the sense that the meta-analysis should analyze each study according to its design, then combine results across studies,” Higgins says. In addition, biases in the included studies, and in the evidence base as a whole, need to be anticipated. “Many meta-analyses probably over-interpret results, but we don’t know if the material is actually biased so it’s difficult to know what to believe. A healthy skepticism is required.”
“A meta-analysis is only as good as the information it gathers, and how that information was gathered,” Etzel says. “Investigators need to be just as stringent on the meta-analysis as they would on any single study. If you produce garbage, you will analyze and report garbage.”
Send us Your Comments
Web Exclusives
- Statistical Advantage
-
Gary Shorter, director of biostatistics for Quintiles, talks to Envisage editor Adam Istas about the growing importance of incorporating biostatistical analysis into all stages of drug development.
- Measure for Measure
-
A proposed Health Care Comparative Effectiveness Research Institute in the United States may force the biopharmaceutical industry to emphasize patient outcomes and quality of life issues when developing a new medicinal product.