Ben Goldacre of badscience.net has released Bad Science, the book, with some inexact and sloppy references (see standard preamble). Goldacre previously highlighted Andrew Taylor’s vaguely systematic and random review of the appropriateness of some of the references and remarked: “This is excellently everything I expect from my readers. Totally bravo”. So, I trust that Goldacre will be as enamoured of my observations although I may have some small differences of opinion about some of the references.
To be fair, the difference lies in the fact that I am reviewing the references from a different perspective. I will stipulate that as far as I can tell, Taylor is correct that Goldacre has reported the crux or results of a paper correctly and appropriately.(a) Given the number of formal references, one substantial omission from the book (aside from the index) is a guide to the general reader as to the best way to search for journal papers in science or medicine journals. Goldacre might have guided interested readers towards Google Scholar or PubMed.
In the spirit of quasi-randomness, I’ve chosen to scrutinise some of the references in Chapter 4: Homeopathy. I’ve selected chapter 4 because it is one of the most informative while being annoyingly discursive and over-abundant in digressions: the errors with the graphs and the messy references rub salt in those paper cuts. I think that are some references that are surprising omissions from Chapter 8: ‘Pill solves complex social problem’.
If you are looking for a journal paper with which you are unfamiliar, a pet peeve is the number of authors who make careless typos in an author’s name that can make a search laborious or even impossible. Although Taylor checked the reference for Schultz [sic] et al, he didn’t notice that the name should be Schulz, if JAMA has the correct list of authors [pg 46 with reference pg 325].
Pg 39 with reference pg 325: this is minor but the Marshall paper refers to NNNT (numbers needed to not treat) rather than NNT (numbers needed to treat), as given in the reference. Some of the additional discussion is a little confusing.(a)
Pg 47 with reference pg 326: or the case of the missing reference (which is something that Taylor’s method was not designed to identify). Goldacre refers to “a bog-standard review of trials for homoepathic arnica by Professor Edward [sic] Ernst” but doesn’t supply a reference. I assume that he meant this review: Ernst E, Pittler MH. Efficacy of homeopathic arnica: a systematic review of placebo-controlled clinical trials. Arch Surg. 1998 Nov;133(11):1187-90.
If the missing paper is Ernst & Pittler it makes sense of the reference to Hildebrandt et al (pg 48) and the footnote studies on pg 51. After introducing this missing review on pg 47, Goldacre discusses randomisation and blinding before introducing two more “landmark studies” (both of which are used correctly) and then returning to nos moutons (Ernst & Pittler) on pg 50.
Pg 52-3 with reference pg 326: pg 52, Goldacre refers to “[t]his graph, from Ernst’s paper” but it doesn’t seem as if it is the Ernst & Pittler cited above; it is probably a different Ernst & Pittler reference that Goldacre cites for the diagram but it isn’t clear and it is impractical to check because it isn’t open access.
Pg 55-6 with reference pg 326: whether or not this is a missing reference is a matter of taste but it might have been useful to have a link to the inaugural Cochrane meta-analysis about antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth.
Pg 144 with reference pg 331: I haven’t been able to locate Goldacre’s summary on badscience.net. I would have preferred these trials to be listed in the references.
There have been…six trials to date on fish oil in children. Not one of these trials was done in ‘normal’ mainstream children: all of them have been done in special categories of children with one diagnosis or another – dysxlexia, ADHD, and so on. Three of the trials had some positive findings…and three were negative. One, amusingly, found that the placebo group did better than the fish-oil group on some measures. They are all summarised online at badscience.net. [pg 144]
I think that there are 2 missing references in chapter 4 but the Cochrane paper may be my preference for completeness. There are several irritating errors and another personal decision as to whether it is acceptable to use “sore throats” and “throat infections” as if they are interchangeable. There are 6 references missing in situ in chapter 8, pg 146, although one of these (the Richardson & Montgomery) is cited on pg 146.
However, compared to Goldacre’s analysis of Holford’s references (Chapter 9: pp 164-70), these errors seem comparatively minor (except for the omissions, especially where they cause confusion in the text). There do not seem to be any instances where the subject, outcome or results of a paper are distorted or misrepresented.(a) More usefully, Goldacre’s references are rather more substantial and traceable than a cassette recording of a long-ago lecture or a wrongly-dated report from the Bateman Catering Organisation.
Goldacre shows no signs of referenciness. Although Goldacre outlines why the works of Gillian McKeith and Patrick Holford should probably come with the warning, caveat lector, his own references appear to be appropriate and accurate (as per Taylor).(b)
I have some reservations that Goldacre has neglected to include some references: some of this seems accidental and sloppy editing; other omissions look like a failure to address a legitimate area of discussion (eg, chapter 5 discusses the placebo effect so it jars that Goldacre does not mention the Cochrane review that concluded that there is little evidence to support ‘a large effect for the placebo response’).
Notes
(a) Although Goldacre refers to a ‘cold’ or ’sore throat’, Marshall discusses “throat infections”: it’s not clear that these terms should be used interchangeably and it puts a slightly different take on why so many patients didn’t change their behaviour. Eg, I was surprised that more patients didn’t accept the antibiotics “are ineffective in treating a viral cold” argument [pg 39 footnote]; however, if the study analysed behaviour relating to throat infections, then it seems more sensible that it might be more difficult for a lay person to distinguish viral and bacterial infections and so some patients would feel the need to consult a GP. To be scrupulously fair, Marshall’s three references all mention “sore throat” in the titles, so Goldacre may be more accurate than Marshall on this point but I don’t believe that it is the reader’s responsibility to check that, plus using different terms can create unnecessary uncertainty.
(b) The caveat here is that I could not check most of the papers, just the ones in which I am interested and that are available online and free of charge. There may be mistakes or infelicities that I don’t know about but so far it looks like he is playing with the “scrupulously straight bat” he advocates when dealing with others (pg 165).
I acknowledge that I can’t comment on the areas that need a good knowledge of the topic in order to be able to judge whether Goldacre’s arguments or conclusions are appropriate.
Related Reading
Tags: Bad Science, Ben Goldacre, inaccuracy, references
