Is the evidence-based medicine movement counter-productive: are randomised controlled trials the best approach to establish evidence in complex healthcare situations? March 2023, Volume 33, Issue 1

Introduction

Evidence-based medicine (EBM) is defined as “a systemic approach to analyse published research as the basis of clinical decision making.”¹ Initially, it was heralded as the new approach to practising medicine, providing clinicians with the tools to critique evidence from research rather than relying on empirical evidence obtained and compounded over decades of historical experience.¹ In its essence, EBM applies population findings to individual patients. However, the process has been criticised for denigrating pathophysiological reasoning and clinical expertise, over-simplifying information, and denying the social contexts in which medicine takes place.¹ Presently, the literature contains publications addressing the role of EBM in patient care, advocating arguments both for and against EBM. Here we assess the role of research that contributes to EBM, the domain that underpins the derivation of ‘evidence’. This paper examines how randomised controlled trial (RCT) study designs fail to balance complex social circumstances. We argue that this contributes to research outcomes returning non-significant findings and may subject participants to unnecessary risks associated with randomisation.

The RCT is considered the gold standard in EBM.² The study design applies a randomisation process to reduce bias and provide a robust, replicable, controlled environment for hypothesis testing within a population.² While many RCTs offer valuable insights on fundamental biological and clinical relationships, RCTs focused on complex disease processes repeatedly report statistically non-significant findings.³ Up to 35% of reanalysed RCTs return findings contrary to those originally reported.³

This paper focuses on RCTs, addressing research questions within complex populations where social and cultural determinants and individual differences are intertwined with the disease process. It builds on previous work outlining limitations of EBM using RCTs in complex health issues,³ which demonstrated that RCTs often produce non-significant results in situations with high complexity because many of the components within the systems being examined are weighted heavily by individuality within patient samples. This leads to the irreproducibility of RCTs, as sample populations are not homogenous. This paper applies research examples derived from ear pathology, more specifically otitis media (OM), as an example to highlight RCT outcomes in complex systems.

Otitis media in remote Aboriginal populations

Otitis media (OM) presents commonly in Australian children.⁴ However, ear pathology is recognised to be less complicated in non-Indigenous and urban-based Aboriginal and Torres Strait Islander populations (hereafter respectfully referred to as Aboriginal) than in Aboriginal populations living in remote areas.⁴ Many clinical factors contribute to OM. While eustachian tube dysfunction is considered the major cause^5,6, other underlying factors can lead to OM including: individual anatomy⁵; individual immunology⁷; the contribution of nutrition, sleep, and the social environment⁶; household number; number sharing a bedroom and cohabiting with active ear or nasal/sinus infections⁷; plus household stress, including: interpersonal violence, substance misuse, and financial stressors.⁸

In remote-living Aboriginal populations, ear pathology presents as extremely complex, with high poverty levels and poor access to health services.^9-11 Yet, despite established evidence for medical and surgical management options for non-Aboriginal populations¹², OM persists in Aboriginal populations. Furthermore, in these populations, routine clinical management of OM reports equal numbers of successes and failures when prescribed the same clinical treatment.¹¹

The following are examples of published findings from studies conducted in remote Aboriginal communities seeking to determine evidence-based best practices to improve ear health. One RCT study on chronic suppurative otitis media (CSOM) compared topical antibiotic eardrops and placebo in Aboriginal children and reported statistically significant findings.^7. The study design adjusted for living conditions, family size, type of housing, prior history and treatment of otorrhoea, and risk factors for CSOM. However, the same RCT study design reported non-significant findings when repeated in a different remote Aboriginal location. ¹³ Another recent RCT for CSOM, Indigenous Healthy EARs — BEtadine, Tissues and Antibiotics study (IHEARBETA), compared 16 weeks of treatment of either a twice daily antiseptic ear wash (using povidone-iodine [0.5%] ear cleaning⁷) or a twice daily oral antibiotic treatment given in addition to standard topical antibiotic treatment (compared to placebo).¹⁴ The outcomes of this study were presented at a conference in Darwin in 2018, and findings indicated no significant difference between treatment arms.¹⁵ One further non-clinical RCT intervention aimed at improving CSOM with the intervention of daily swimming in chlorinated swimming pools. Once again, no significant difference in CSOM was found between children with pool access and children who engaged in non-swimming play activities.¹⁶

The evidence for best-practice surgical options for OM with effusion (OME) are limited to a Cochrane systematic review with meta-analyses on tympanostomy tube insertion, which demonstrated improved hearing three months after tube insertion (12dB better), modest improvement in hearing; 4 decibels 6–  months after tube insertion and no difference in hearing (treatment versus control) 12–18 months after insertion.¹⁷  There was no effect demonstrated on language, speech or cognitive development or quality of life outcomes.¹⁷ A systematic review with meta-analyses is considered National Health and Medical Research Council (NHMRC) Level I evidence² and individual RCTs are considered Level II evidence, while case-series evidence is considered Level IV evidence.²

An RCT is presently being conducted, funded by NHMRC, comparing outcomes between three treatment groups: 1) no surgery, medical intervention only; 2) adenoid removal and myringotomy; and 3) adenoid removal plus ‘grommet’ (ventilation tube) insertion. Although authors of this NHMRC study recently recommended that the best surgical management for OME includes tympanostomy tubes or ‘grommets’ for Indigenous and non-Indigenous children¹⁸, they also indicate that “despite the availability of evidence-based guidelines, giving treatment advice is a challenge because recommendations vary according to condition, age, risk of complications and parental preference”.¹⁸

In ear health, as with other areas of research, the harms associated with conducting the research are rarely published. Yet it is possible that harm could be associated with the use of using povidone-iodine ear wash as a treatment for CSOM¹⁴, which has long been established as ototoxic in animals and has recently been demonstrated as ototoxic in a human case report.¹⁹ For surgical interventions in remote tropical Aboriginal communities, the intervention usually requires grommet insertion, with potential harms including chronic aural discharge, a common complication arising after getting ears wet, and especially prevalent if wetness is due to contaminated water.⁶ Therefore, randomising children to grommet insertion, when it is common practice to swim in local creeks and muddy water, is likely to introduce pathogens and cause further complications. ⁶ Hence, children randomised to the grommet arm of the study are both less likely to obtain successful research outcomes and may incur further middle ear complications.⁶

RCT study design limitations

So, despite decades of research, this case study suggests there remains little evidence for determining OM best practice for Aboriginal populations in remote areas. Almost all studies report non-significant outcomes, despite rigorous and well-balanced RCT study designs. One explanation is that the factors driving ear disease in remote Aboriginal communities are complex and not solely due to the presence of pathogens; this ‘complexity’ constitutes the ‘unknown’ factors within the study design.

When clinical RCTs are undertaken in complex environments, it is often difficult to design a study that adjusts for all the elements involved. Most RCTs are designed with the assumption that the main outcome variable – the variable targeted by the research question or treatment intervention – is driving the problem or the disease state, such as a bacterial pathogen. Yet, in complex populations living in complex environments, such as remote Aboriginal populations, it is highly likely the predictor variable plays a lesser role in driving the disease process, and the ‘unknown’ variables are more influential. Known contributing factors may be controlled within the RCT study design. However, the study design cannot control for unknown contributors. In the context of remote Aboriginal communities, unknown contributing factors may be sensitive or difficult to source for legal or social reasons. These can include household interpersonal violence²⁰, substance misuse, or financial stressors.⁸ These frequently unmeasured contributors are typically not incorporated into RCT study designs. We suggest that it is unlikely they will ever be, as ethics committees would be reluctant to support their unfettered inclusion due to privacy and the potential risk of self-incrimination.

Another driver of disease that is unaccounted for in RCT study designs within complex populations are the social hierarchies and kinship ties within these communities, often adding nuance to poverty measurement beyond that which can be detected even by Indigenous-specific socioeconomic indicators.²¹ These hierarchies and kinship ties provide important social cohesion and social structure²¹ and are accepted within the community. Yet, at times they foster the development and advantage of fellow kin at the expense of non-kin.²² Access to socioeconomic improvements such as new houses and employment are among the advantages available to fellow-kin within the community hierarchy.²² These types of advantages can impact overall community health, yet they are highly unlikely to be incorporated into study designs.

Can we improve OM management without RCTs?

The most compelling evidence on successful strategies to improve ear health in remote Aboriginal populations is to improve underlying living standards, healthcare access, and health literacy.²³ Investment in Child Hearing Health Co-ordinator program that applies a case management model of service delivery has demonstrated marked improvements to the hearing health of Aboriginal children in the Northern Territory, Australia.²³ While this program is not a research study, its evaluation, and reporting provide evidence that the best way to achieve successful outcomes in ear health is for local health workers to work closely with expert clinicians, who tailor individual care options for each child and their family. In many cases, the resulting treatment plan may still not deliver successful outcomes, but it offers the best available care. Research in this instance isn’t necessary, as the best practice has been established. Furthermore, the established best practice doesn’t place the child at risk of harm through research participation, which may not position the child in the best study arm for their circumstance. Perhaps the best research option is ‘service delivery with evaluation’, which does not employ the RCT, but still applies a research framework to provide evidence and is less likely to place the participant at risk.²⁵

It is likely that the most significant changes to remote Aboriginal ear health will only occur with a corresponding rise in the standard of living, regardless of the amount of investment in ear health research. This increase in the standard of living requires more than healthcare investment and is certainly beyond the scope of any biomedical RCT. Further investment in patient-centred primary health care is likely to improve outcomes faster and more ethically than RCTs.

In Australia, the highest standard of EBM evidence is obtained when a systematic review is conducted on all available RCTs within a defined scope, considered Level I evidence (NHMRC)², with case-series evidence considered Level IV evidence. However, in the UK, the Oxford Centre for Evidence-Based Medicine applies additional levels under Case-series Level IV evidence²⁵, which includes first principles and expert opinion as Level D evidence. In complex systems, the best available evidence would likely be Level D, as there is little compelling evidence at the higher levels. So, one policy implication might be for the NHMRC to put greater value on evidence from lower levels in complex circumstances. The NHMRC could also consider evidence from retrospective studies or audits that compare current clinical practices with historical ones. These study designs do not require individual consent to an intervention arm and are less likely to be detrimental to participants. Lastly, human research ethics committees play a role in recommending caution when endorsing clinical RCTs if these studies involve established complex health environments. This is especially pertinent when previous studies have been undertaken and returned non-significant findings time and time again.

Conclusions

Clinical RCTs addressing complex health issues where social determinants play a major contributing role are unlikely to return meaningful results. In these circumstances, the continual attempts to conduct RCTs deplete community goodwill and increase scepticism in the research process. It is preferable that clinicians acknowledge that the RCT is unlikely to assist in decision-making, and the best treatment options for complex conditions are tailored treatments to individual patients when the clinician balances multiple pathophysiological variables and the social situation of each individual. While treatments may not always return a successful outcome, this remains the best outcome for that patient. We believe that the tide may turn, and society will move toward trusting expert clinical knowledge as empirical evidence, alongside collaboration with colleagues as smart medicine, and move away from the current obsession with RCT-based EBM, which continues to drive unsuccessful research.

Acknowledgements

We acknowledge Dr Mark Wenitong for reviewing this manuscript before submission.

Peer review and provenance

Externally peer reviewed, not commissioned.

 

Copyright:

© 2023 Jacups and Bradley. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.