Extracted from the Submission to the Commonwealth Ombudsman by Complementary Medicines Australia, Australian Homoeopathic Association and Australian Traditional Medicine Society, submitted in August 2016. HRI provided the scientific analysis for this submission.
Between October 2010 and March 2015 the National Health and Medical Research Council (NHMRC) conducted an investigation into homeopathy, to inform the Australian community on the “effectiveness of homeopathy.” This investigation was an example of NHMRC’s function to ‘advise the community’ under section 7(1)(a) of the NHMRC Act 1992. It was instigated under the NHMRC Strategic Plan 2010-2012 “to examine the evidence underlying the alternative medicines most highly used” and culminated in a formal review of the evidence for effectiveness of homeopathy (the Homeopathy Review).
NHMRC’s findings were that, for the 61 health conditions covered by the Review:
‘…no good-quality, well-designed studies with enough participants for a meaningful result reported either that homeopathy caused greater health improvements than placebo, or caused health improvements equal to those of another treatment’.
Their overall conclusion, based on this assessment of the evidence was that, “…there are no health conditions for which there is reliable evidence that homeopathy is effective.”
NHMRC’s media release announcing publication of the Review claimed that, “The conclusion is based on the findings of a rigorous assessment of more than 1800 papers” and was accompanied by a statement presenting their interpretation of the results as, “Homeopathy should not be used to treat health conditions that are chronic, serious, or could become serious.”
The potential impact of an evidence review by a respected institution such as NHMRC cannot be overstated – the general public, health practitioners, decision-makers and other researchers all rely on their findings. It is therefore essential that NHMRC reviews are free from bias, providing a fair and objective assessment of a given topic. Risk of bias is normally minimised by three key safeguards:
- Use of standardised and accepted scientific methods
- Internal policies and procedures g. adherence to NHMRC legislation, standards, guidelines and conflict of interest policy
- Transparency and accountability g. public disclosure of processes followed, meaningful public consultation and accurate communications to the public.
It is our contention that in the case of the NHMRC Homeopathy Review all three safeguards were breached, exposing the process to unacceptable levels of anti-homeopathy bias – evident in both its administrative and methodological aspects – which led directly to distortion of the Review results. Key examples of this bias are outlined below:
In December 2010 the NHMRC agreed to a draft position statement, describing homeopathy as ‘unethical’, ‘inefficacious’, ‘implausible’ and even ‘deceptive’. This statement was prepared without any scientific evaluation of homeopathy by NHMRC, being based solely on the findings of a single non-academic report prepared by a committee of Members of Parliament in the UK. This anti-homeopathy position on the part of NHMRC was echoed in an article published around the same time by its CEO Professor Anderson who described homeopathy as “a retreat from reason” and “an alleged therapy”. In the wake of criticism relating to bias and lack of procedural and scientific rigour following a leak of this draft statement into the public domain, NHMRC initiated a formal investigation into homeopathy, instigated by Professor Anderson.
Professor Anderson appointed an expert overview committee for the Review that was compromised by conflicts of interest and bias from the outset. This Homeopathy Working Committee (HWC), which was directly involved in deciding how the evidence was analysed and interpreted, was initially chaired by Professor Peter Brooks who signed a Declaration of Interest (DOI) form, declaring he was not “affiliated or associated with any organisation whose interests are either aligned with or opposed to homeopathy,” despite being a member of the anti-homeopathy medico-political lobby group Friends of Science in Medicine (FSM). The CEO appointed Professor Brooks as Chair after being notified by letter of FSM’s attempt to influence NHMRC reviewers. After the conflict was exposed, although Professor Brooks stood down as Chair, the CEO/NHMRC supported his continuing presence as an active member of the HWC for the duration of the Review.
The HWC also failed to contain a single expert in either homeopathy or homeopathic research; exclusion of a topic expert is unprecedented in NHMRC processes of this kind and is in breach of NHMRC guidelines and policies informing appointments to its expert committees.
The HWC carried out the formal review of the evidence for effectiveness of homeopathy twice, under two different external contractors. NHMRC’s claim that the first review, which appears to have found positive evidence for the effectiveness of homeopathy, was rejected on grounds of poor quality, not its results; this is despite the review being conducted by a highly experienced and reputable reviewer who is co-author of NHMRC’s own Additional Levels of Evidence and Grades for Recommendations for Developers of Guidelines guidance document. The existence of this first review (final draft completed in August 2012) was hidden from the public and NHMRC continues to refuse to disclose details of this review and why it was terminated.
The method used for assessing the evidence the second time (resulting in the final published Homeopathy Review), was not a ‘standardised accepted method’; it was created specifically for this Review by NHMRC. The approach hinged on their unique definition of ‘reliable’ evidence i.e. for a trial to be ‘reliable’ it had to have more than 150 participants and meet an unusually high standard of trial quality (scoring 5 out of 5 on the ‘Jadad’ quality rating scale or equivalent on other scales). All trials which fell below either of these thresholds were disregarded as being of ‘insufficient size and/or quality to warrant further consideration of their findings’ (Overview Report, p.38). NHMRC’s quality threshold for a ‘reliable’ trial is highly unusual, but their decision to set a minimum ‘sample size’ of 150 participants for trial reliability was completely arbitrary, unprecedented and cannot be justified scientifically.
When explaining their decision to classify trials with less than N=150 as being ‘small’ or ‘very small’, NHMRC refer to an article in the highly respected journal BMJ1. NHMRC said: “HWC considered the following study in the development of these thresholds: Influence of trial sample size on treatment effect estimates: meta-epidemiological study.” (Overview Report Appendices, p.274). This implies that their decision to dismiss trials smaller than N=150 as ‘unreliable’ is scientifically justified by this paper. It is not. NHMRC correctly describes most of the homeopathic trials they assessed as using ‘continuous outcomes’, yet the BMJ paper states categorically that its findings cannot be applied to trials of this kind.
NHMRC used this citation of the BMJ study against the N=150 threshold multiple times across the final report documents released to the public, who would not question that an expert body such as NHMRC would make such a fundamental error and/or intentionally publish misleading information.
In NHMRC’s 2015 media release they further misled the public by stating that “more than 1800” papers underwent “rigorous assessment”, implying that their conclusions were based on a thorough examination of this exhaustive body of evidence. In fact, although 1863 papers on homeopathy were identified by NHMRC reviewers or submitted by external parties, NHMRC’s choice of inclusion criteria meant that only 267 studies were assessed in any detail, being considered to be possibly relevant to the Review; of those, only 176 trials were finally assessed as suitable to be entered into the Review.
Applying NHMRC’s combined ‘reliability’ filter of 150 participants plus very high quality, then led to 171 out of 176 trials being dismissed as ‘unreliable’ and of “insufficient size and/or quality to warrant further investigation of their findings”. Having reduced the evidence base to only 5 ‘reliable’ trials, none of which NHMRC considered to show homeopathy to be effective, the HWC found (unsurprisingly) that there is ‘no reliable evidence’ that homeopathy is effective.
The direct impact of this approach was to exclude good quality, positive trials showing homeopathy to be effective, thus distorting the results. This issue was identified in 2013 by NHMRC’s own expert independent reviewer with no connection to the homeopathy sector, who expressed concern about NHMRC’s concluding statements, saying, “If the intent is to provide general statements about the effectiveness of homeopathy, then ‘no reliable evidence’ may not adequately reflect the research. For example, when a substantial proportion of small (but good quality) studies show significant differences, […] ‘no reliable evidence’ does not seem an accurate reflection of the body of evidence.”2
NHMRC’s chosen scientific method for the Homeopathy Review was an ‘overview’ i.e. a ‘review of reviews’. This meant that instead of analysing the 176 individual studies themselves, they relied only on secondary data provided about those trials in other ‘systematic reviews’ (SRs) that summarise evidence. As much of the necessary information was inaccurate or missing, this inherent flaw reduces the credibility of the Review.
A further ‘limitation’ listed by NHMRC in using this approach is that they may have missed relevant single trials that were not described in the SRs, but misleadingly suggest that this ‘risk’ was ‘offset’ by inviting submissions from homeopathy interest groups and a via a formal public consultation. They inaccurately state that this externally submitted evidence was, “…assessed using a similar method to that applied in the overview” (Information Paper, p.8) but ‘… did not alter the overall findings of the assessment of the evidence (Information Paper, p.25).