Are we reaching a turning point? There have been several papers recently drawing attention to problems with the poor design of many animal experiments. Some of them are briefly summarised below.
If it is now recognised that there is a problem, should we be getting on to the next step, of doing something about it?
But there is a dilemma. Better experimental design means fewer positive (but false) results. That may mean fewer exciting papers in high impact journals, leading to less funding for those labs which are doing things well. Take, for example, the Scott et al paper (see below). More than 50 papers have been published on drugs which appeared to increase lifespan in a mouse model of ALS, but only one works clinically. Scott et al suggest that virtually all of these are false positives. They have now screened 70 compounds using a much better experimental design and improved statistical methods, and all are negative. After five years of careful work, they have no exciting positive papers to publish. Yet their work is clearly of great value.
Below are comments or extracts from several recent papers
Funding organisations promote use of ARRIVE Guidelines (August 2012).Heads of three major funding organisation (MRC, BBSRC and Wellcome Trust), write an open letter to university vice-chancellors, Principals and Heads of Divisions encouraging scientists to improve the design of experiments using animals and use the ARRIVE guidelines. Click here for the press release.
The ‘3Is’ of animal experimentation (June 2012)
“Raise standards for preclinical cancer research “ (March 2012)
“Beware the creeping cracks of bias” (May 2012)
An example of bias from conflicts of interest (2005)
Beware of conflicts of interest. 94/98 government funded studies found that bisphenol A (BPA), a compound widely used in the production of rigid plastics, caused developmental abnormalities in rats or mice whereas 0/8 industry funded studies found an effect. This was after subtracting the 13 studies done using the CD:SD stock, which is resistant to the effects of diethylstilboestrol and similar compounds.
Presumably any government funded research which failed to find an effect could not now be published as the authors would appear to be incompetent, and industry is unlikely to publish any studies showing a positive effect. Clinical studies are now under way, although these must be cross-sectional or cohort studies rather than experiments, which would be unethical. BPA has been banned from use in the manufacture of baby’s bottles.
Design, power and the interpretation of studies in the standard murine model of ALS (2008)
More than 50 papers have described therapeutic agents which extend the lifespan of SOD1G93A transgenic mice (the standard model of ALS) but only one of these, riluzole has any clinical effect. So why were all these drugs so ineffective in humans?
Using computer modelling and statistical analysis Scott et al identified a number of confounding factors (e.g. gender, litter, censoring, copy number) likely to influence the outcome of these experiments and used a power analysis to develop an optimum experimental design for such studies. They then screened 70 drugs (including those claimed as “positive” in previous tests) using 18,000 mice in 221 studies over a five-year period using rigorous statistical techniques. None of them (including riluzole) extended the lifespan of the mice.
The identification of these uncontrolled confounding factors and the failure of the drugs to have any effect in well controlled repeated experiments led them to conclude that most of the published effects are simply the result of noise rather than drug effects. (These noise effects would be exacerbated by any failures in randomisation and blinding).