News

27 July, 2020

Second time lucky? Timeline of unreported changes to the research protocol, NHMRC homeopathy review


Post Image

A key undertaking provided by the Department of Health for the new 2019-20 review updates of 16 natural therapies is to pre-publish the research protocols to be used. This is an important transparency measure routinely observed in ethical scientific processes to safeguard them against research malpractice and bias. 

This decision has a historical basis. In the previous 2012-15 reviews of 16 natural therapies undertaken by the National Health & Medical Research Council (NHMRC), the research protocols were not pre-published and the reports released in 2015 did not provide full disclosure. Why?

This article provides a detailed timeline, for the public record, of post-hoc changes made to the research protocol for the first natural therapy reviewed in 2012-15 (homeopathy), and how these changes impacted the findings. The approach developed for this review provided the template for how the other 15 natural therapies were assessed. For further background to the genesis of these reviews, read here.

These details are included in a Complaint being investigated by the Commonwealth Ombudsman of alleged misconduct in the 2015 NHMRC Homeopathy Review, which is expected to conclude soon.

The problem of research malpractice:

Misconduct/fraud in clinical research is recognised as a widespread problem and is surprisingly common in the conduct of health-related systematic reviews [1]. The NHMRC recognises this issue and states that it takes all research integrity matters very seriously. According to the NHMRC Fraud Control Framework 2020-22:

“NHMRC refuses to tolerate fraud and has a commitment to high ethical, moral and legal standards.”

Two of the commonest features of research malpractice are the failure to follow the agreed research protocol (including making midstream changes in the absence of full disclosure and justification) and inadequate/inaccurate reporting of methods and results [2].

The importance of research protocols:

Before a study begins, a protocol (the investigation plan) is created which outlines in detail all essential aspects of the project, such as the research question being asked, methods of data retrieval, criteria used to determine which studies will be included or excluded from the review, and how the included data will be analysed to produce the final results.

Making significant ‘post-hoc’ (midstream) changes is a recognised source of bias as the reviewers may, consciously or subconsciously, alter the method to achieve a pre-desired result. Any changes made to the protocol must be fully disclosed and justified.

This is why protocols are often published before a study begins so that once the final study is published, other researchers can see whether the original protocol was followed correctly and independently assess any post-hoc changes that were made. It is an important safeguard protecting against fraudulent conduct that undermines the scientific process and public trust.

PROSPERO:

To counter such problems, in February 2011 the open-access database International Prospective Register of Systematic Reviews (PROSPERO) was launched for research groups to prospectively register reviews on health-related topics.

“PROSPERO aims to provide a comprehensive listing of systematic reviews registered at inception to help avoid duplication and reduce opportunity for reporting bias by enabling comparison of the completed review with what was planned in the original protocol.”

In the 2019-20 Natural Therapy Review, the Department of Health has informed stakeholders that the NHMRC will prospectively register the research protocol for each natural therapy with PROSPERO, for transparency and to safeguard against bias.

PROSPERO only accepts reviews provided that data extraction has not yet started, “to reduce potential for bias by reducing the opportunity for (conscious or subconscious) selection or manipulation of data during extraction to shape a review so that it reaches a desired conclusion.”

NHMRC and the PROSPERO advisory group:

PROSPERO is supported and guided by an international advisory group that includes the NHMRC’s Prof Davina Ghersi, who was also involved in the establishment of PROSPERO in late 2010/2011 [3]. This was over a year before the NHMRC homeopathy review formally commenced on 4 April 2012.

The Freedom of Information (FOI) record shows that Prof Ghersi was also involved in the NHMRC homeopathy review during 2012 and 2013, alongside the Chair of NHMRC Homeopathy Working Committee (HWC), Bond University’s Prof Paul Glasziou. She liaised with the contractors, was a key contributor at HWC meetings, and was directly involved in developing the methodology.

The missing first review:

Documents released under FOI in 2016 revealed that the NHMRC engaged in two attempts at reviewing the evidence on homeopathy. The report released on 11 March 2015, which concluded there was ‘no reliable evidence’, was in fact the second attempt.

The existence of a first review conducted between April and August 2012 by the University of South Australia (UniSA), which reported positive findings for homeopathy in some conditions, was not disclosed to taxpayers. Since the existence of the review was not made public, the pre-agreed research protocol was also not known.

FOI documents show that on 12 July 2012, HWC member Prof Fred Mendelsohn provided the following advice to NHMRC on a draft version of the UniSA report [4],

“I am impressed by the rigour, thoroughness and systematic approach given to this evaluation … Overall, a lot of excellent work has gone into this review and the results are presented in a systematic, unbiased and convincing manner.”

The contractor submitted their final draft report on 1 August 2012 and “after discussions with the contractor on 3 August” the NHMRC terminated their contract [5]. After initially considering completing the work themselves [5], the NHMRC instead decided to engage a new contractor, OptumInsight (Optum), and start again. The 2015 NHMRC Administrative Report (p.6) states that Optum was commissioned in October 2012 to review the evidence – without disclosing any of the preceding events.

Following is a general overview of what happened during the second attempt. This is followed by an in-depth, fully referenced timeline for those who wish to acquaint themselves with the ‘behind the scenes’ details of what occurred during the Optum review. This information is published in the public domain for transparency and accountability, as it was not disclosed or reported by the NHMRC in over 944 pages of published documentation.

Overview of undisclosed changes to the Optum review research protocol:

The (first) UniSA and (second) Optum reviews reached different conclusions on the basis of essentially the same data set. How did the Optum review conclude there was ‘no reliable evidence’, whereas the UniSA review concluded there was ‘encouraging evidence’ in several conditions?

The FOI record shows that the research protocol for the Optum review was finalised and approved between all parties (NHMRC, HWC and Optum) in late December 2012 [6] and that Optum completed its assessment between January and March 2013 according to it specifications [7].

The protocol was not prospectively published on a registry such as PROSPERO (or equivalent) or ever made publicly available.

Since the approved protocol was not published, how closely Optum followed it was not able to be determined, however, the minutes of the first HWC meeting held in March 2018 to discuss the findings show that the approved protocol was not fully followed [7].

FOI documents reveal that the Office of NHMRC/ HWC then convened a Sub-group from April 2013 to ‘further refine’ the protocol [8]. FOI records show that over the coming months, a new framework was created around a unique concept of ‘reliable evidence’, which continued to be modified and refined until August 2013. The final version, which bore no resemblance in any respect to the protocol originally approved in December 2012, was an entirely novel framework that had never been used before (or since) by the NHMRC or any other research group or agency in the world.

None of the post-hoc changes made to the original protocol, or their impact on the findings, were disclosed/reported.

Impact of post-hoc protocol changes on review’s findings:

Independent assessment of NHMRC’s method and procedures post-publication of the 2015 report has shown that the changes made directly resulted in 171 out of the 176 studies included in the Optum overview being retrospectively categorised as ‘not reliable’. This meant that the results of these studies were ‘not considered any further’ and therefore did not contribute to the review’s findings.

Even though these 171 studies were technically included in the review and described in the Optum Overview Report, their results were dismissed from the review’s findings as ‘unreliable’. This important analysis was not included in the Optum Overview Report or in any NHMRC documentation.

Since the 5 remaining ‘reliable’ trials were judged to be negative, the NHMRC reported an overall finding of ‘no reliable evidence’ in any of the 61 conditions examined.

The timeline of retrospective, undisclosed changes made to the research protocol during the 2012-15 NHMRC homeopathy review (as revealed by the FOI record) is summarised in Figure 1 below:

Figure 1: Timeline of changes to NHMRC (Optum) Homeopathy Review research protocol & process used for drafting evidence statements (‘ES framework’) *For details of protocol changes see main text.

Second time lucky? A detailed timeline of post-hoc changes to the research protocol

The NHMRC has defended allegations that it manipulated the research protocol during the second (Optum) review in any way that altered the findings, or that it misleadingly reported its methods and findings. This has been referred to the Commonwealth Ombudsman for independent review. The NHMRC states that the review was conducted ‘ethically’ and ‘transparently’, citing its reputation for excellence, integrity and position of public trust.

Read the following story and judge for yourself.

December 2012 – Optum review research protocol finalised and approved:

Documents released under FOI in 2016 show that the original research protocol for the Optum Review was finalised and approved by the NHMRC, Optum and the HWC in late December 2012, prior to Optum commencing the work [6].

The final approved protocol, which was never published, revealed that the following ‘standardised’ framework was to be applied by Optum when developing evidence statements against each health condition assessed [6]:

Research protocol approved for Optum Review, Dec 2012

Of particular note is that this evidence statement framework protocol bears no resemblance, in any respect, to the one published in the NHMRC Information Paper (Appendix C) and Optum Overview Report (Appendix C).

It is also noteworthy that the protocol approved in December 2012 allows for uncertainty in the findings, which is common in systematic reviews, especially in areas (such as homeopathy) where a large body of evidence is not available.

So how did the protocol change, when did this occur and were these changes and their impact on the findings reported?

Detailed timeline of post-hoc changes to the research protocol:

January-March 2013 – Optum completes evidence assessment:

After the original research protocol was finalised and approved in December 2012, Optum conducted its data extraction and evidence assessment between 3 January and March 2013. The HWC then convened on 18 March 2013 to discuss the findings and consider the draft evidence statements [7].

The minutes of the March 2013 HWC meeting show that when developing draft evidence statements, the original protocol was not followed because it would have found that the evidence for homeopathy was “uncertain” (i.e. not negative) [7]. From this point, the FOI record shows that a decision was made to deviate from the originally approved protocol in order to alter the results of the review to reach more definitive conclusions, as described below.

Apr-May 2013 – HWC Sub-group established to ‘refine’ the research protocol:

To do this, the NHMRC established a HWC Sub-group, which convened between 8 April and 24 May 2013 for the purpose of ‘refining’ the evidence statement framework [8]. The existence of the Sub-group process or its actions was not disclosed in the Optum Overview Report, NHMRC Information Paper or Administrative Report, or elsewhere.

At the first Sub-group meeting on 8 April, none of the criteria used in the final evidence statement framework yet existed [9]. All elements of the final framework were developed and retrospectively applied to the assessment over the coming months, as part of an iterative process where unreported changes were made to the protocol along the way (Fig. 1).

8 April 2013 – inaugural Sub-group meeting:

The minutes of the inaugural Sub-group meeting on 8 April record the intention to redraft the pre-agreed evidence statement protocol. The group reaffirmed the purpose of the evidence statements, “to reflect the body of evidence, rather than drawing conclusions about the effectiveness of homeopathy”. This was later changed to make them ‘conclusive’ (see below, Fig. 1).

The minutes of the 8 April meeting show how the group grappled with developing evidence statements for conditions where positive, good quality positive evidence existed for homeopathy (such as childhood diarrhoea and influenza), as well as how the process itself was deviating from the pre-approved protocol [9]:

29 April 2013 – ‘null hypothesis approach’ first proposed:

The 2015 NHMRC Information Paper (Appendix C, p.38) states that the ‘null hypothesis approach’ was applied to the evaluation, meaning it was ‘assumed’ that “homeopathy has no effect as a treatment for that condition unless there was sufficient reliable evidence to demonstrate otherwise”.

What the Information Paper doesn’t mention is that this was not part of the original protocol, but first introduced by the HWC Chair (Prof Paul Glasziou) midstream in late April 2013 [10, 11], weeks after Optum had completed its assessment. The revision was formally adopted at the 29 April HWC Sub-group meeting. An email from the NHMRC to the HWC on 30 April obtained under FOI shows this change, and the awareness that it represented a midstream change in approach [10]:

It is also important to note use of the term ‘reliable‘ evidence – what does this mean?

By this stage of the review (late April/ early May 2013), the HWC/ NHMRC/ Optum themselves had no idea what it meant, as the criteria/ elements that comprised the ‘reliable evidence’ framework had not yet been developed. In fact, even the concept of ‘reliable evidence’ did not yet formally exist.

The FOI record shows that this framework was developed over the following months, gradually sculpted into its final form along the way.

29 Apr 2013 – ‘adapted GRADE’ tool & ‘conclusive statements’ introduced:

FOI documents reveal that at the 29 April Sub-group meeting, Prof Davina Ghersi (NHMRC) first proposed the use of the ‘GRADE’ tool to provide a ‘level of confidence (LOC)’ rating of the evidence [11]. This was to eventually become ‘Element 2‘ of the final evidence statement framework.

Optum stated that it was “unfamiliar with the GRADE tool” and was “reluctant to apply it without a full understanding of how it should be used”. Therefore, Prof Ghersi undertook to develop a novel ‘adapted’ version of it specifically for the review, and that she and/or the HWC would apply it on Optum’s behalf [11]:

The ‘need’ for introducing “conclusive statements” as a new element of the evidence statement framework was also first introduced. This was a fundamental shift from the original protocol agreed in December 2012, which specified that the purpose of the review was to “inform the community of the evidence” and “not draw conclusions” (allowing for the uncertainty that invariably exists in research evidence). ‘Conclusive statements’ was introduced as a new element (Element 3) of the revised, still incomplete protocol.

At the 29 April meeting, the idea of categorising studies by their ‘size’ was also first flagged as a concept. An action item was for Prof Ghersi to consider and develop this idea further between meetings for the HWC’s later consideration.

It was also noted that the NHMRC would need to “consult with the HWC to get their approval for the proposed revised approach”, showing the key role the NHMRC played in driving the process (with the FOI record showing the primary role that the HWC Chair, Prof Paul Glasziou, also played).

The minutes of the 29 April meeting record, “any agreed approach would then be applied going forwards”, acknowledging that the protocol was being modified post-hoc with a view to applying it retrospectively [11].

6 May 2013 – HWC approves revised draft evidence statement framework

The new ‘revised’ framework was approved by the HWC at its teleconference meeting on 6 May 2013, including the new ‘adapted GRADE’ tool [12].

The meeting minutes record discussion on how the framework could be applied to a number of conditions for which good quality, positive evidence existed (such as diarrhoea in children, otitis media, ADHD, fibromyalgia, influenza-like illness), which the first (UniSA) reviewer had framed as ‘encouraging’ evidence.

The NHMRC/HWC grappled with developing consistent statements for these conditions, so the NHMRC undertook to draft ‘example statements’ for these conditions for the HWC to consider at later meetings, and to refine the protocol further as required [12].

24 May 2013 – ‘sample size’ threshold concept first introduced:

The 2015 NHMRC Information Paper (Appendix C, p.35) states that one of the criteria for a trial to be ‘reliable’ is that it had to have a minimum of 150 trial participants.

What it doesn’t reveal is that the concept of using trial sample size as an exclusion threshold did not exist until late May 2013, and that when it was first conceived it was not linked to any concept of trial ‘reliability’ [13].

The FOI record shows that at the 24 May 2013 HWC Sub-group meeting, Prof Ghersi (NHMRC) first proposed a sample size threshold of 200 participants for whether a trial was ‘adequately powered’. This was introduced as a new component of Element 1 of the (still evolving) evidence statement framework [13].

9 July 2013 – First round methodological peer review, ACC:

The NHMRC Information Paper (p.15) states, “NHMRC commissioned an independent organisation with expertise in research methodology (The Australasian Cochrane Centre) to review the methods used in the overview and ensure that processes for identifying and assessing the evidence were scientifically rigorous, consistently applied, and clearly documented.”

NHMRC did not publish the ACC’s advice, which was obtained through FOI [14]. On 9 July 2013, the ACC peer-reviewed the draft report and advised NHMRC/HWC:

– that linking sample size to whether a trial is ‘adequately powered’ was not scientifically valid

– of fundamental flaws with the novel ‘adapted GRADE’ tool. This is because GRADE is designed to be used when describing primary research evidence, whereas the NHMRC review only accessed secondary sources (systematic reviews) that omitted and/or incompletely reported essential primary source information (which is not their purpose).

– that the overly-definitive draft evidence statements did not allow for uncertainty in the research (which the original protocol approved in December 2012 would have allowed for).

11-12 July 2013 – HWC face-to-face meeting, revised sample size exclusion threshold approved:

As a result of the ACC’s advice, the NHMRC/ HWC dropped the minimum N=200 sample size threshold for whether a trial was ‘adequately powered’ and replaced it with a new minimum threshold of N=150 sample size for whether a trial was ‘reliable’.

At its 11-12 July 2013 face-to-face meeting, the HWC agreed to the new N=150 sample size exclusion threshold for trial ‘reliability’, which was added to Element 1 of the evidence statement framework [15]. This is when the concept of ‘reliable evidence’ first appears.

The impact of this decision was momentous in the context of the review, but this is not possible to see from the NHMRC report as this data was not provided anywhere in over 900 pages of report documentation. It is noteworthy that NHMRC routinely conducts studies with less than 150 participants, which are not classed as ‘unreliable’.

Impact of the N=150 rule – independent evaluation:

Independent evaluation has shown that in its own right, the N=150 rule to determine whether a trial was ‘reliable’ dismissed the results of 146 out of the 176 included trials from ‘being considered any further’ as part of the review’s findings [16].

The fact that the criterion was introduced four months after Optum had completed its evidence assessment (seven months after the original protocol was approved) was also not reported.

Late July/ August 2013 – ‘quality rating scale’ threshold added:

Having eliminated the results of 146 of the 176 identified trials from contributing to the review’s findings, 30 ‘reliable’ trials remained by virtue of having more than 150 participants.

At some stage in late July/ August 2013, the NHMRC/HWC introduced a further ‘quality rating’ element. This first appears in the FOI record in late August 2013 [17].

For a trial to be considered ‘reliable’, it now also had to be rated an unusually high 5/5 on the Jadad (or equivalent) quality rating scale to be considered ‘good quality’, and therefore ‘reliable’. This new criterion was newly added as a component of Element 1 of the evidence statement framework.

This was not reported as a change to the protocol and its impact was also not reported.

Independent evaluation [16] has shown that the ‘quality rating’ criterion directly dismissed the results of 25 out of the remaining 30 trials from ‘being considered any further’ as part of the Review’s findings.

One in four trials “assumed” to be substandard quality, without checking original papers:

The NHMRC also did not report that for 44 of the 176 trials (25% of the data), their ‘quality’ could not be determined as the secondary-source information (systematic reviews) relied upon didn’t report it. Instead of retrieving the original studies to check, the NHMRC/HWC invented a ‘rule’ to assume they were ‘not good quality’ and therefore ‘not reliable’ (see NHMRC Information Paper pp.35-36).

This left only 5 ‘reliable’ trials, all of which were deemed to be negative, facilitating an overall conclusion that there was ‘no reliable evidence that homeopathy worked in any health condition’.

Remaining ‘reliable’ trial:

One of the 5 remaining ‘reliable’ trials was in fact positive – a ‘good’ quality trial (5/5 Jadad) on lower back pain with over 150 participants. In Table 1 of the NHMRC Information Paper, this trial is substituted with a negative trial in the same condition that was not included as part of the main overview. An error?

FAQ: ‘All 176 trials seem to be included in the NHMRC overview, not just 5 trials’

The NHMRC media release of 11 March 2015 misleadingly announced that the results of its review were based on a “rigorous assessment of over 1800 studies” [18]. While Optum’s initial data extraction identified over 1800 papers (which included several hundred duplicate citations), only 176 trials were accepted for inclusion in the overview.

The NHMRC Information Paper correctly states that these 176 studies were described and evaluated in the Optum Overview Report. In this respect, they were ‘included’ in the overview.

However, if a trial did not meet the minimum ‘N=150 participants’ and/or ‘5/5 Jadad (or equivalent) quality rating’ thresholds, they were deemed to not be ‘reliable’ and their results (whether positive, negative or inconclusive) were dismissed as ‘not warranting further consideration of their findings’. As the NHMRC Information Paper itself confirms (on p.36, at the back of the report in Appendix C):

“If there was more than one study that suggested that homeopathy is more effective than placebo or as effective as other therapies but due to the number, size and/or quality of those studies the findings are not reliable, a general statement to that effect was made, for example: ‘These studies are of insufficient [quality] / [size] / [quality and size] / [quality and/or size] / [quality or size] to warrant further consideration of their findings.”

Neither the Information Paper nor any other documentation disclosed/reported that this dismissed the results of 171 out of the 176 studies from contributing to the findings.

30 August 2013 – Second round methodological peer review, ACC:

The ACC peer-reviewed the revised draft report and made a number of observations and recommendations that were not reported or followed. The ACC advised, “There are instances where our initial concerns about a particular methodological approach stand” (in particular the ‘adapted GRADE’ tool) and also noted that post-hoc changes to the research protocol had been made, which had not been fully disclosed or adequately explained.

For example, the ACC noted that ‘The document now specifies cutoffs for applying qualitative descriptors of quality for different tools (p2). …’ (which did not exist in the draft report it peer-reviewed in July 2013). ACC advised, “Empirical evidence has shown that using quality scales to identify trials of high quality is problematic”.

The ACC also critiqued the overly-definitive nature of the review’s findings, which it noted was not consistent with the existence of a significant proportion of small, but good quality studies. It advised NHMRC/HWC (emphasis added):

“If the intent is to provide general statements about the effectiveness of homeopathy, then ‘no reliable evidence’ may not adequately reflect the research. For example, when a substantial proportion of small (but good quality) studies show significant differences, […] ‘no reliable evidence’ does not seem an accurate reflection of the body of evidence.”

The FOI record shows that the HWC dealt with this feedback by changing the purpose of the evidence statements, from ‘informing‘ the community of the evidence (the purpose of the review, as framed in the original protocol) to, advise members of the community about the effectiveness of homeopathy to support informed healthcare decisions/ choices. This retrospective, change to the framing of the evidence statements, by giving it a more directive (‘advisory’) context, was a nuance that appeared to ‘solve the problem’ in the eyes of the NHMRC/HWC.

The 9 July and 30 August ACC feedback was not disclosed, despite the NHMRC releasing a dedicated ‘Expert Review Comments’ document alongside the Information Paper.

Additional undisclosed peer-review comments, 2014:

The NHMRC ‘Expert Review Comments’ [19] document also heavily redacted adverse feedback provided by another peer-reviewer in May 2014, after the draft report was released for public comment on 9 April 2014. This expert reviewer’s feedback was obtained through FOI and revealed similar concerns expressed to NHMRC, which were disregarded and kept out of the public eye.

In the following expert reviewer’s feedback to NHMRC, the text highlighted in red was excluded from the NHMRC ‘Expert Review Comments’ document [20]:

“The dismissal of positive systematic reviews compounded with the lack of an independent systematic review of high quality randomised controlled trials leaves me uncertain of the definitive nature of the Report’s conclusions. […]

High quality RCTs with narrow confidence intervals (Level 1 evidence) should have been searched for and included in this review.

Systematic reviews (SRs) have considerable weaknesses as reliable sources of evidence. Personally, I would prefer a much more reserved approach to their use as Level 1 evidence. For example, we know that SRs can come to quite contrasting conclusions pending the grading RCT scale they adopt. (See Juni et al, JAMA, 1999). High quality clinical trial designs in homeopathy should accommodate for tailoring of treatment but this is generally not weighted in the rating scales used by reviewers. If tailoring of treatment is critical in homeopathy then it may be that only low quality studies, as defined by the common grading metrics, will exhibit positive outcomes. [This is because tailoring of treatment may have resulted in un-blinding of the intervention group.] Some systematic reviews conclude homeopathy is more than placebo (Cucherat et al 2000; Linde & Melchart 1998; Linde et al 1997; Kleijnen 1991; and many of the reviews in the Swiss report found a trend in favour of homeopathy). It is probably unreasonable to discount this evidence on the basis that good quality trials did not show such strong evidence of efficacy, if the quality rating scale for trials is not well justified for use in homeopathy. Hence, the usual conclusion that high quality trials demonstrate less favourable outcomes than poor quality trials does not really hold.

Quality rating scales have seldom been tested for inter-rater reliability, there is a lack of agreement between scales as to what is being measured, and scales differ in many respects including items for inclusion and number of items. The NHMRC Report should address this issue more explicitly. […]

… if I were to dispassionately consider the evidence of efficacy, I am still left with niggling doubts that there are unanswered questions around the evidence. […] Finally, I am concerned that no homeopathic expert was appointed to the NHMRC Review Panel. I cannot imagine this being agreed in oncology, orthopaedics or other disciplines.”

NHMRC policy on research integrity:

The NHMRC, as the peak medical research agency in Australia, creates the rules that it expects all medical researchers to follow and states that it, “takes all research integrity matters very seriously“.

All research funded by the NHMRC is required to comply with the Australian Code for the Responsible Conduct of Research, 2018, as part of the NHMRC Research Integrity and Misconduct Policy.

 

Sources:

[1] Elia N, von Elm E, Chatagner A, et al (2016). How do authors of systematic reviews deal with research malpractice and misconduct in original studies? A cross-sectional analysis of systematic reviews and survey of their authors. 

[2] Gupta A. (2013). Fraud and misconduct in clinical research: A concern. Perspectives in clinical research4(2), 144–147. https://doi.org/10.4103/2229-3485.111800

[3] Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., Stewart, L. (2012). The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Syst Rev 1, 2. https://doi.org/10.1186/2046-4053-1-2

[4] 12 Jul 2012. Feedback to NHMRC from HWC member Prof Fred Mendelsohn on the July 2012 version of the first (UniSA) reviewer’s Draft Report. NHMRC FOI 2014/15 021-08 & NHMRC FOI 2016/17 016-Doc 13

[5] 13 Aug 2012 – Email between Office of NHMRC & HWC Chair Paul Glasziou re termination of UniSA contract. NHMRC FOI 2014/15 021-Doc 10

[6] 21 Dec 2012. Email from NHMRC to Optum & HWC members with final research protocol for the Optum Homeopathy Review. NHMRC FOI 2014/15 004-Section 58

[7] 18 March 2013. HWC face-to-face meeting minutes. NHMRC FOI 2015/16 007-Doc 3

[8] Minutes of HWC and HWC Sub Group meetings 2013-2014. NHMRC FOI 2015/16 008

[9] 8 Apr 2013. Minutes of inaugural HWC Sub-group teleconference meeting to discuss evidence statements. NHMRC FOI 2015/16 008-Doc 1

[10] 30 Apr 2013. Email from ONHMRC to HWC members re. ‘null hypothesis’ approach to homeopathy evidence statements. NHMRC FOI 2014/15 021-Doc 11

[11] 29 Apr 2013. HWC Sub Group teleconference meeting minutes. NHMRC FOI 2015/16 008-Doc 2

[12] 6 May 2013. HWC Sub Group teleconference meeting minutes. NHMRC FOI 2015/16 008-Doc 3

[13] 24 May 2013. HWC Sub Group teleconference meeting minutes. 24.05.2013. NHMRC FOI 2015/16 008-Doc 7

[14] 9 Jul 2013. First-round Australasian Cochrane Centre (ACC) methodological peer review. NHMRC FOI 2015-16 008-Doc 13

[15] 11-12 Jul 2013. HWC face-to-face meeting minutes. NHMRC FOI 2015-16 008-Doc 6

[16] Homeopathy Research Institute (HRI). Data extracted from NHMRC’s Table 1 (Information Paper, p.18- 20), with analysis of the combined impact of ‘reliability’ thresholds for trial sample size and quality. https://www.hri-research.org/wp-content/uploads/2017/07/HRI-data-analysis-impact-of-NHMRC-definition-of-reliable.pdf

[17] 30 Aug 2013. Australasian Cochrane Centre (ACC) second-round methodological peer review. NHMRC FOI 2015-16 007-Doc 5

[18] 11 Mar 2015. NHMRC Releases Statement and Advice on Homeopathy (media release)

[19] 11 Mar 2015. Summary of key issues: Draft information paper on homeopathy—expert review comments. NHMRC cam02e. Available at: https://www.nhmrc.gov.au/about-us/resources/homeopathy

[20] 10 May 2014. NHMRC Expert Reviewer feedback re. Draft Information paper on homeopathy. NHMRC FOI 2014/15 004-Section 62

 

 

SIGN below to support scientific integrity and hold government processes to account.


« Return to News & Features
Background Image

SIGN HERE

Are you one of the 2 in 3 people that use traditional medicine and natural therapies to complement your health needs?

Add your voice to the growing community of people that wish to protect their right to access these therapies. Let’s send Government a clear message.

116027

people signed

We’ll keep you updated about the campaign. For more information please see our About us page.