Advanced search
Perspectives

Is the evidence-based medicine movement counter-productive: are randomised controlled trials the best approach to establish evidence in complex healthcare situations? March 2023, Volume 33, Issue 1

Susan P Jacups, Clare Bradley

Published 15 March 2023. https://doi.org/10.17061/phrp3312303
Citation: Jacups SP, Bradley C. Is the evidence-based medicine movement counter productive: are randomised controlled trials the best approach to establish evidence in complex healthcare situations? Public Health Res Pract. 2023;33(1):e3312303.

  • Citation

  • PDF

About the author/s

Susan P Jacups | School of Public Health, University of Queensland, Brisbane, Australia; The Cairns Institute, James Cook University, Queensland, Australia.

Clare Bradley | School of Public Health, University of Queensland, Brisbane, Australia; South Australian Health and Medical Research Institute, Adelaide, Australia; College of Medicine and Public Health, Flinders University, Adelaide, SA, Australia

Corresponding author

Susan P Jacups | [email protected]

Competing interests

None declared.

Author contributions

SJ was responsible for the concept, design, drafting, and editing of the manuscript. CB added to the concept, and also edited the manuscript.

Abstract

In the modern era, evidence-based medicine (EBM) has been embraced as the best approach to practising medicine, providing clinicians with ‘objective’ evidence from clinical research. However, for presentations with complex pathophysiology or from complex social environments, sometimes there remains no evidence, and no amount of research will obtain it. Yet, health researchers continue to undertake randomised controlled trials (RCT) in complex environments, ignoring the risk that participants’ health may be compromised throughout the trial process.

This paper examines the role of research that seeks to obtain evidence to support EBM. We provide examples of RCTs on ear disease in Aboriginal populations as a case-in-point. Decades of ear research have failed to yield statistically significant findings, demonstrating that when multiple factors are at play, study designs struggle to balance the known disease process drivers, let alone unknown drivers. This paper asks the reader to consider if the pursuit of research is likely to produce evidence in complex situations; or if perhaps RCTs should not be undertaken in these situations. Instead, clinicians could apply empirical evidence, tailoring treatments to individuals while taking into account the complexities of their life circumstances.

Full text

Key points

  • Since the evidence-based medicine (EBM) approach to practising medicine commenced, research undertaken to obtain ‘evidence’ has focused on randomised controlled trials (RCTs), which often yield non-statistically significant results
  • Researchers may not recognise the impact of unmeasurable contributions to the disease process in complex social environments, which may be driving the disease. These contributions create bias in the RCT study design
  • We use the case study of ear disease in Aboriginal populations to argue that when undertaking research in complex environments, lower level ‘evidence’, such as retrospective studies – which pose less risk to participant safety – should be supported

Introduction

Evidence-based medicine (EBM) is defined as “a systemic approach to analyse published research as the basis of clinical decision making.”1 Initially, it was heralded as the new approach to practising medicine, providing clinicians with the tools to critique evidence from research rather than relying on empirical evidence obtained and compounded over decades of historical experience.1 In its essence, EBM applies population findings to individual patients. However, the process has been criticised for denigrating pathophysiological reasoning and clinical expertise, over-simplifying information, and denying the social contexts in which medicine takes place.1 Presently, the literature contains publications addressing the role of EBM in patient care, advocating arguments both for and against EBM. Here we assess the role of research that contributes to EBM, the domain that underpins the derivation of ‘evidence’. This paper examines how randomised controlled trial (RCT) study designs fail to balance complex social circumstances. We argue that this contributes to research outcomes returning non-significant findings and may subject participants to unnecessary risks associated with randomisation.

The RCT is considered the gold standard in EBM.2 The study design applies a randomisation process to reduce bias and provide a robust, replicable, controlled environment for hypothesis testing within a population.2 While many RCTs offer valuable insights on fundamental biological and clinical relationships, RCTs focused on complex disease processes repeatedly report statistically non-significant findings.3 Up to 35% of reanalysed RCTs return findings contrary to those originally reported.3

This paper focuses on RCTs, addressing research questions within complex populations where social and cultural determinants and individual differences are intertwined with the disease process. It builds on previous work outlining limitations of EBM using RCTs in complex health issues,3 which demonstrated that RCTs often produce non-significant results in situations with high complexity because many of the components within the systems being examined are weighted heavily by individuality within patient samples. This leads to the irreproducibility of RCTs, as sample populations are not homogenous. This paper applies research examples derived from ear pathology, more specifically otitis media (OM), as an example to highlight RCT outcomes in complex systems.

Otitis media in remote Aboriginal populations

Otitis media (OM) presents commonly in Australian children.4 However, ear pathology is recognised to be less complicated in non-Indigenous and urban-based Aboriginal and Torres Strait Islander populations (hereafter respectfully referred to as Aboriginal) than in Aboriginal populations living in remote areas.4 Many clinical factors contribute to OM. While eustachian tube dysfunction is considered the major cause5,6, other underlying factors can lead to OM including: individual anatomy5; individual immunology7; the contribution of nutrition, sleep, and the social environment6; household number; number sharing a bedroom and cohabiting with active ear or nasal/sinus infections7; plus household stress, including: interpersonal violence, substance misuse, and financial stressors.8

In remote-living Aboriginal populations, ear pathology presents as extremely complex, with high poverty levels and poor access to health services.9-11 Yet, despite established evidence for medical and surgical management options for non-Aboriginal populations12, OM persists in Aboriginal populations. Furthermore, in these populations, routine clinical management of OM reports equal numbers of successes and failures when prescribed the same clinical treatment.11

The following are examples of published findings from studies conducted in remote Aboriginal communities seeking to determine evidence-based best practices to improve ear health. One RCT study on chronic suppurative otitis media (CSOM) compared topical antibiotic eardrops and placebo in Aboriginal children and reported statistically significant findings.7. The study design adjusted for living conditions, family size, type of housing, prior history and treatment of otorrhoea, and risk factors for CSOM. However, the same RCT study design reported non-significant findings when repeated in a different remote Aboriginal location. 13 Another recent RCT for CSOM, Indigenous Healthy EARs — BEtadine, Tissues and Antibiotics study (IHEARBETA), compared 16 weeks of treatment of either a twice daily antiseptic ear wash (using povidone-iodine [0.5%] ear cleaning7) or a twice daily oral antibiotic treatment given in addition to standard topical antibiotic treatment (compared to placebo).14 The outcomes of this study were presented at a conference in Darwin in 2018, and findings indicated no significant difference between treatment arms.15 One further non-clinical RCT intervention aimed at improving CSOM with the intervention of daily swimming in chlorinated swimming pools. Once again, no significant difference in CSOM was found between children with pool access and children who engaged in non-swimming play activities.16

The evidence for best-practice surgical options for OM with effusion (OME) are limited to a Cochrane systematic review with meta-analyses on tympanostomy tube insertion, which demonstrated improved hearing three months after tube insertion (12dB better), modest improvement in hearing; 4 decibels 6–  months after tube insertion and no difference in hearing (treatment versus control) 12–18 months after insertion.17  There was no effect demonstrated on language, speech or cognitive development or quality of life outcomes.17 A systematic review with meta-analyses is considered National Health and Medical Research Council (NHMRC) Level I evidence2 and individual RCTs are considered Level II evidence, while case-series evidence is considered Level IV evidence.2

An RCT is presently being conducted, funded by NHMRC, comparing outcomes between three treatment groups: 1) no surgery, medical intervention only; 2) adenoid removal and myringotomy; and 3) adenoid removal plus ‘grommet’ (ventilation tube) insertion. Although authors of this NHMRC study recently recommended that the best surgical management for OME includes tympanostomy tubes or ‘grommets’ for Indigenous and non-Indigenous children18, they also indicate that “despite the availability of evidence-based guidelines, giving treatment advice is a challenge because recommendations vary according to condition, age, risk of complications and parental preference”.18

In ear health, as with other areas of research, the harms associated with conducting the research are rarely published. Yet it is possible that harm could be associated with the use of using povidone-iodine ear wash as a treatment for CSOM14, which has long been established as ototoxic in animals and has recently been demonstrated as ototoxic in a human case report.19 For surgical interventions in remote tropical Aboriginal communities, the intervention usually requires grommet insertion, with potential harms including chronic aural discharge, a common complication arising after getting ears wet, and especially prevalent if wetness is due to contaminated water.6 Therefore, randomising children to grommet insertion, when it is common practice to swim in local creeks and muddy water, is likely to introduce pathogens and cause further complications. 6 Hence, children randomised to the grommet arm of the study are both less likely to obtain successful research outcomes and may incur further middle ear complications.6

RCT study design limitations

So, despite decades of research, this case study suggests there remains little evidence for determining OM best practice for Aboriginal populations in remote areas. Almost all studies report non-significant outcomes, despite rigorous and well-balanced RCT study designs. One explanation is that the factors driving ear disease in remote Aboriginal communities are complex and not solely due to the presence of pathogens; this ‘complexity’ constitutes the ‘unknown’ factors within the study design.

When clinical RCTs are undertaken in complex environments, it is often difficult to design a study that adjusts for all the elements involved. Most RCTs are designed with the assumption that the main outcome variable – the variable targeted by the research question or treatment intervention – is driving the problem or the disease state, such as a bacterial pathogen. Yet, in complex populations living in complex environments, such as remote Aboriginal populations, it is highly likely the predictor variable plays a lesser role in driving the disease process, and the ‘unknown’ variables are more influential. Known contributing factors may be controlled within the RCT study design. However, the study design cannot control for unknown contributors. In the context of remote Aboriginal communities, unknown contributing factors may be sensitive or difficult to source for legal or social reasons. These can include household interpersonal violence20, substance misuse, or financial stressors.8 These frequently unmeasured contributors are typically not incorporated into RCT study designs. We suggest that it is unlikely they will ever be, as ethics committees would be reluctant to support their unfettered inclusion due to privacy and the potential risk of self-incrimination.

Another driver of disease that is unaccounted for in RCT study designs within complex populations are the social hierarchies and kinship ties within these communities, often adding nuance to poverty measurement beyond that which can be detected even by Indigenous-specific socioeconomic indicators.21 These hierarchies and kinship ties provide important social cohesion and social structure21 and are accepted within the community. Yet, at times they foster the development and advantage of fellow kin at the expense of non-kin.22 Access to socioeconomic improvements such as new houses and employment are among the advantages available to fellow-kin within the community hierarchy.22 These types of advantages can impact overall community health, yet they are highly unlikely to be incorporated into study designs.

Can we improve OM management without RCTs?

The most compelling evidence on successful strategies to improve ear health in remote Aboriginal populations is to improve underlying living standards, healthcare access, and health literacy.23 Investment in Child Hearing Health Co-ordinator program that applies a case management model of service delivery has demonstrated marked improvements to the hearing health of Aboriginal children in the Northern Territory, Australia.23 While this program is not a research study, its evaluation, and reporting provide evidence that the best way to achieve successful outcomes in ear health is for local health workers to work closely with expert clinicians, who tailor individual care options for each child and their family. In many cases, the resulting treatment plan may still not deliver successful outcomes, but it offers the best available care. Research in this instance isn’t necessary, as the best practice has been established. Furthermore, the established best practice doesn’t place the child at risk of harm through research participation, which may not position the child in the best study arm for their circumstance. Perhaps the best research option is ‘service delivery with evaluation’, which does not employ the RCT, but still applies a research framework to provide evidence and is less likely to place the participant at risk.25

It is likely that the most significant changes to remote Aboriginal ear health will only occur with a corresponding rise in the standard of living, regardless of the amount of investment in ear health research. This increase in the standard of living requires more than healthcare investment and is certainly beyond the scope of any biomedical RCT. Further investment in patient-centred primary health care is likely to improve outcomes faster and more ethically than RCTs.

In Australia, the highest standard of EBM evidence is obtained when a systematic review is conducted on all available RCTs within a defined scope, considered Level I evidence (NHMRC)2, with case-series evidence considered Level IV evidence. However, in the UK, the Oxford Centre for Evidence-Based Medicine applies additional levels under Case-series Level IV evidence25, which includes first principles and expert opinion as Level D evidence. In complex systems, the best available evidence would likely be Level D, as there is little compelling evidence at the higher levels. So, one policy implication might be for the NHMRC to put greater value on evidence from lower levels in complex circumstances. The NHMRC could also consider evidence from retrospective studies or audits that compare current clinical practices with historical ones. These study designs do not require individual consent to an intervention arm and are less likely to be detrimental to participants. Lastly, human research ethics committees play a role in recommending caution when endorsing clinical RCTs if these studies involve established complex health environments. This is especially pertinent when previous studies have been undertaken and returned non-significant findings time and time again.

Conclusions

Clinical RCTs addressing complex health issues where social determinants play a major contributing role are unlikely to return meaningful results. In these circumstances, the continual attempts to conduct RCTs deplete community goodwill and increase scepticism in the research process. It is preferable that clinicians acknowledge that the RCT is unlikely to assist in decision-making, and the best treatment options for complex conditions are tailored treatments to individual patients when the clinician balances multiple pathophysiological variables and the social situation of each individual. While treatments may not always return a successful outcome, this remains the best outcome for that patient. We believe that the tide may turn, and society will move toward trusting expert clinical knowledge as empirical evidence, alongside collaboration with colleagues as smart medicine, and move away from the current obsession with RCT-based EBM, which continues to drive unsuccessful research.

Acknowledgements

We acknowledge Dr Mark Wenitong for reviewing this manuscript before submission.

Peer review and provenance

Externally peer reviewed, not commissioned.

 

Copyright:

Creative Commons License

© 2023 Jacups and Bradley. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.

 

References

  • 1. Claridge JA, Fabian TC. History and development of evidence-based medicine. World J Surg. 2005;29(5):547—53. CrossRef | PubMed
  • 2. National Health and Medical Research Council (NHMRC) additional levels of evidence and grades for recommendations for developers of guidelines. Sydney: Med J Aust; 2009 [cited 2023 Jan 17]. Available from: www.mja.com.au/sites/default/files/NHMRC.levels.of.evidence.2008-09.pdf
  • 3. Keane M, Berg C. Evidence-based medicine: a predictably flawed paradigm. Trends in Anaesthesia and Critical Care 2016;9:49–52. CrossRef
  • 4. Kong K, Coates HL. Natural history, definitions, risk factors and burden of otitis media. Med J. Aust. 2009;191 (9 Suppl):S39–43. CrossRef | PubMed
  • 5. Schilder AGM, Bhutta MF, Butler CC, Holy C, Levine LH, Knaerner KJ, et al. Eustachian tube dysfunction: consensus statement on definition, types, clinical presentation and diagnosis. Clin Otolaryngol 2015;40(5):407–11. CrossRef | PubMed
  • 6. O’Leary SJ, Triolo RD. Surgery for otitis media among Indigenous Australians. Med J Aust. 2009;191(S9):S65–S8. CrossRef | PubMed
  • 7. Couzos S, Lea T, Mueller R, Murray R, Culbong M. Effectiveness of ototopical antibiotics for chronic suppurative otitis media in Aboriginal children: a community-based, multicentre, double-blind randomised controlled trial. Med J Aust. 2003;179(4):185–90. CrossRef | PubMed
  • 8. Rabkin JG, Struening EL. Life events, Stress, and Illness. Science. 1976;194(3):1013–20. CrossRef | PubMed
  • 9. Jacups SP, Kinchin I, McConnon KM. Ear, nose, and throat surgical access for remote living Indigenous children: what is the least costly model? J Eval Clin Pract. 2018;24(6):1330–8. CrossRef | PubMed
  • 10. Jacups SP, McConnon KM. Reducing ear, nose and throat (ENT) waitlists: implications of a referral audit. Health Policy 2019;123(3):333–7. CrossRef | PubMed
  • 11. Coleman A, Wood A, Bialasiewicz S, Ware RS, Marsh RL, Cervin A. The unsolved problem of otitis media in indigenous populations: a systematic review of upper respiratory and middle ear microbiology in indigenous children with otitis media. Microbiome. 2018;6(1):199. CrossRef | PubMed
  • 12. Leach AJ, Morris PS, Coates HL, Nelson S, O'Leary SJ, Richmond PC, et al. Otitis media guidelines for Australian Aboriginal and Torres Strait Islander children: summary of recommendations. Med J Aust. 2021 Mar;214(5):228–33. Available from: www.health.gov.au/sites/default/files/documents/2021/07/otitis-media-guidelines.pdf
  • 13. Leach A, Wood Y, Gadil E, Stubbs E, Morris P. Topical ciprofloxin versus topical framycetin-gramicidin-dexamethasone in Australian aboriginal children with recently treated chronic suppurative otitis media: a randomized controlled trial. Pediatr Infect Dis J. 2008;27(8): 692–8. CrossRef | PubMed
  • 14. Wigger C, Leach AJ, Beissbarth J, Oguoma V, Lennox R, Nelson S, et al. Povidone-iodine ear wash and oral cotrimoxazole for chronic suppurative otitis media in Australian Aboriginal children: study protocol for factorial design randomised controlled trial. BMC Pharmacol Toxicol. 2019;20(1):46. CrossRef | PubMed
  • 15. Wigger C. Preliminary clinical treatment outcomes of a randomised controlled trial using Betadine® ear wash and oral Bactrim® for chronic suppuratives otitis media in Aboriginal Children. Otitis Media Australia Conference (OMOZ) 2018. Darwin, Australia: Menzies School of Health Research; 2018. Abstract available from authors.
  • 16. Stephen AT, Leach AJ, Morris PS. Impact of swimming on chronic suppurative otitis media in Aboriginal children: a randomised controlled trial. Med J Aust. 2013;199(1):51–5. CrossRef | PubMed
  • 17. Browning GG, Rovers MM, Williamson I, Lous J, Burton MJ, Williamson I. Grommets (ventilation tubes) for hearing loss associated with otitis media with effusion in children. Cochrane Database Syst Rev. 2010(10):CD001801. CrossRef | PubMed
  • 18. Kong K, Lannigan FJ, Morris PS, Leach AJ, O'Leary SJ. Ear, nose and throat surgery: All you need to know about the surgical approach to the management of middle-ear effusions in Australian Indigenous and non-Indigenous children. Paediatr Child Health. 2017;53(11):1060–4. CrossRef | PubMed
  • 19. Piromchai P. Ototoxicity of povidone-iodine – a case report. J Otol. 2019;14(1):30–2. CrossRef | PubMed
  • 20. Krug EG, Mercy JA, Dahlberg LL, Zwi AB. The world report on violence and health. Lancet 2002;360 (9339):1083–8. CrossRef | PubMed
  • 21. O’Brien K. Social cohesion and resilience in First Australian family and kinship networks. Journal of Family History 2017;42(4):440–51. CrossRef
  • 22. McRae-Williams E, Gerritsen R. Mutual incomprehension: the cross cultural domain of work in a remote Australian Aboriginal community. International Indigenous Policy Journal. 2010;1(2). CrossRef
  • 23. Closing the Gap Clearinghouse. Ear disease in Aboriginal and Torres Strait Islander children. Canberra: Australian Institute of Health and Welfare & Melbourne: Australian Institute of Family Studies; 2014 [cited 2023 Jan 17]. Available from: www.aihw.gov.au/getmedia/c68e6d27-05ea-4039-9d0b-a11eb609bacc/ctgc-rs35.pdf.aspx
  • 24. Kong K, Cass A, Leach AJ, Morris PS, Kimber A, Su J-Y, et al. A community-based service enhancement model of training and employing Ear Health Facilitators to address the crisis in ear and hearing health of Aboriginal children in the Northern Territory, the Hearing for Learning Initiative (the HfLI): study protocol for a stepped-wedge cluster randomised trial. Trials. 2021;22(1):403. CrossRef | PubMed
  • 25. Morsels of Evidence. Evidence-based medicine for general practitioners. Levels of evidence. Australia: Morsels of Evidence: 2019 [cited 2023 Jan 17]. Available from: https://evidencebasedmedicine.com.au/?page_id=30