Filling the gap between evidence, policy and practice: are 45 and Up Study researchers planning for impact? March 2023, Volume 33, Issue 1

Introduction

Researchers have long been producers and disseminators of research, aiming to achieve benefits for society that lie beyond academia. The potential impact of research on policy and practice is broad and far-reaching. The impact of research for the wider population can be observed through the implementation of evidence-informed health policy that can improve people’s quality of life and save lives: the Sabin vaccine for polio, measles vaccination, and smoking cessation programs to name a few.

Hundreds of billions of dollars are spent annually on research and development – with an estimated A$23.1 billion spent in 2018 in Australia alone.¹ However, a consistent finding from clinical and health services research is that research is not translated into policy and practice in a timely manner.^2,3 This resulting evidence-to-practice gap exposes patients to inappropriate care and unnecessary treatments, resulting in high costs and resource wastage.⁴ More broadly, policymakers and program managers report that research is often not timely, relevant or actionable, and opportunities to integrate research into decision making are missed.

Gaps between evidence and practice have been attributed to multiple factors. Policy makers cite a lack of timely, relevant research, its limited applicability to policy contexts, and insufficient organisational access to academic journals as barriers to using evidence in their policies.^5,3 The discordance between evidence and practice may lead to research wastage, e.g. research questions that are irrelevant to decision makers, clinicians and patients, poorly designed studies, biased or unusable reports of research and failing to describe interventions adequately.^6,2

Researchers say that institutional priorities lie with academic metrics, such as journal articles published and conferences attended and often limit access to relationship-building with non-academic actors.⁷ They are not given the time, training, or incentive to effectively engage with policy makers and communicate the information that policy makers require.^8,9 Furthermore, research evidence is identified as only one of many factors that influence policy, with others including policy makers’ values, other sources of information, political ideologies, stakeholder interests, policy drivers, and personal experience.¹⁰

Large longitudinal cohort studies, such as the Sax Institute’s 45 and Up Study, can play a significant role in providing high-quality evidence for decision making relating to policy and practice.¹¹ Cohort studies and linked data are one way of generating research that is relevant to policy makers and practitioners. They provide a powerful tool for understanding the changes in health, health behaviours and the living circumstances of the population over time; the interactions and cumulative effects of factors that affect their health; and the real-world impact of population-level programs and policies.¹² Using existing datasets such as data registries, linked databases, and population cohorts is an efficient way to investigate research questions because they enable analyses of multiple data points.¹³ These data have the potential to generate timely, relevant research that may directly impact the health of the population it seeks to improve.

Linked data from the 45 and Up Study is accessible to researchers, government and nongovernmental bodies¹² and was collected to provide high-quality data to generate policy-relevant information.¹¹ More than 660 researchers have used data from the study since it commenced in 2006.¹¹ This is the first examination of researchers’ intended policy or practice impact, where the impact was achieved and how researchers achieved it. A better understanding of how researchers currently think about and plan for policy and practice impact will help identify strategies that support researchers to do this.

The aims of this study were to identify: 1) the proportion of researchers who intended to have an impact and achieved impact, 2) the types of impact they intended and achieved and 3) the pathways through which they achieved it.

A detailed understanding of researchers’ practice will assist in developing strategies to promote policy and practice impacts and the ways they can be measured in research projects such as the 45 and Up Study.

Methods

Study design

This study is a document review of research project applications, progress reports and final reports submitted by project investigators as part of the 45 and Up Study mandatory reporting requirements.

It compared application forms (where the researchers specified their intended impact) with progress and final reports (where the researchers identified the impact of their work).

The UNSW Human Research Ethics Committee determined that ethics approval was not required for this document review.

Data collection

The data were de-identified by the 45 and Up Study administrators and provided for this document review. Research projects were included in the study if they commenced and were completed between 1 January 2011 and 31 December 2017, where forms were uniform, used a standard format, and included sufficient documentation for analysis (i.e. an application form and one or more progress or final reports). Data on self-reported impact were extracted at three stages in the reporting period: when researchers submitted application, progress, and final reports. The application and progress report data were both tick boxes (in the categories given) and free text (e.g. to provide more details about the expected or final impact of research). Data were categorised to capture policy and practice impact aspects, as detailed in Appendix A, available from figshare.com/articles/journal_contribution/Appendix_A_pdf/19710931.

A coding schedule was developed based on recent work in understanding research use and impact from researchers and funding organisations.^10,14-18. Data on the nine National Health Priority Areas (NHPAs) addressed by the research projects were also extracted, to focus on research areas that Australia has prioritised.¹⁹

The coding schedule defined and categorised the data by type of policy or practice impact, method of achieving research impact (‘pathway’), and target audience (see Appendix A, available from figshare.com/articles/journal_contribution/Appendix_A_pdf/19710931). Following Cohen et al.¹⁵, the coding schedule differentiated between academic outputs or outcomes and those related to policy and practice. Unlike Cohen, we included proximal indicators in our definition of policy or practice change (i.e. any indication that the researchers sought to change, influence, inform, develop or evaluate policy or practice). Under policy and practice impact, we included categories that focused on changes or development to policies and programs, clinical practice or processes, clinical guidelines, health service delivery organisations systems, commercial arrangements and legislation.

Under pathways to impact (how policy or practice impact would be achieved), we included meetings and interaction with policy makers, co-designing research, lobbying and advocacy, journal article publications and conference presentations, teaching and training, plain English resources, discussion papers and reports, media and social media.

Target audiences (with whom findings would be shared) included ministers and ministerial officers, policy makers and program managers, hospital or health service administrators, health practitioners or clinicians, patients or families, consumer organisations, and journalists.

Investigators EF and GM developed and pilot-tested the schedule. To ensure further robustness and reduce partiality, a drift process was conducted in which a random sample of 10% of the research projects EF coded were re-coded by GM. An inter-rater reliability score of 90% was achieved.

Data analysis

Intended impact data were collected and analysed from application forms and were treated as baseline data, as they were submitted at the beginning of the research project. Impact data were identified from progress and final reports and considered as impact achieved.

Data were analysed using the Statistical Package for the Social Sciences (SPSS) version 25 (IBM, 2017). Descriptive statistics and cross-tabulations were used to create tables and figures.

Results

Twenty-eight projects initially met the inclusion criteria. On further examination, three of these studies were excluded as one project was outside the inclusion dates (finishing after 2017), one only provided an application report, and one only provided a progress report. A total of 25 research projects were included in this study.

Project applications addressed 11 research areas with multiple responses possible, resulting in 36 total responses regarding the research areas covered by projects. Just over half (n = 19, 53%) of total responses included an NHPA: cancer control (n = 8, 22%); mental health (n = 3, 8%); obesity/physical activity (n = 3, 8%); cardiovascular health (n = 3, 8%); asthma and respiratory conditions (n = 1, 3%); and arthritis and musculoskeletal health (n = 1, 3%). No other NPHAs were identified in the project applications. Non-NHPA research areas identified include smoking (n = 2, 6%), alcohol use (n = 1, 3%), ageing (n = 1, 3%), autoimmune disorders (n = 1, 3%) and gender (n = 1, 3%). Some projects addressed health systems issues such as preventable hospitalisations and waiting times (n = 4, 11%), and others included a focus on broader influences such as socioeconomic status (n = 3, 8%), housing (n = 2, 6%) and health inequalities (n = 1, 3%). One study examined statistical methods (n = 1, 3%).

Twenty-two (88%) project applications mentioned an intention to achieve one or more policy or practice impacts, with respondents able to nominate an intention to achieve impacts in multiple areas. Of all responses regarding types of intended policy and practice impacts (n = 29), 52% (n = 15) planned to change a policy or program; followed by health service delivery (n = 6, 21%); a clinical practice or process (n = 5, 17%); a clinical guideline or manual (n = 2, 7%); or an organisational system or process (n = 1, 3%) (see Table 1).

Table 1.    Researchers’ intentions to achieve policy or practice impact from 45 and Up Study research projects

^a Respondents could provide multiple responses

Intended pathways for achieving impact are presented in Table 1. Of the 22 research projects which intended to have impact, 10 (45%) did not identify a pathway for achieving impact. Of those who did report a pathway (n = 12), 25% (n = 3) intended to use academic means only (i.e. publishing in academic journals and teaching via seminars and webinars). A further 25% (n = 3) intended to use policy-oriented activities only (i.e. discussion papers and plain English summaries, co-designing research, and interaction with policy makers or advocating for change), and the remaining 50% intended to use a combination of academic and policy-oriented methods.

Of the 22 research projects which mentioned an intended impact, the majority (n = 14, 64%) did not nominate a specific intended audience (see Table 1). The remaining eight projects identified one or more audiences from the types of intended audience, with a total of 15 responses. These included policy makers or program managers (n = 3, 20%), patients and families (n = 3, 20%), health practitioners or clinicians (n = 2, 13%), hospital or health service administrators (n = 2, 13%); and a government minister (n = 1, 7%). The following audiences were nominated in the ‘other’ option: non-government (n = 2), ‘stakeholders’ (n = 1) and the organisation conducting the research (n = 1). (see Table 1). No project mentioned journalists, consumers or consumer organisations as an audience.

Table 2.     Types of output from research projects (N = 25)

Forty-four percent (n = 11) of all the research projects studied reported publishing at least one journal article, and 56% (n = 14) described planning at least one journal article. Nearly one-quarter of the research projects reported neither preparing nor publishing a journal article. Fewer projects reported other outputs: 28% of all 25 projects (n = 7) reported conducting a conference presentation, 28% (n = 7) reported engaging with the media about their research (e.g. news articles, radio mentions, social media posts) and 12% (n = 3) research projects reported publishing other materials (e.g. reports, book chapters) (see Figure 1).

Of the 22 research projects that intended to have a policy or practice impact, four (18%) reported achieving an impact (see Figure 2). Four reported changes to policy, and one also reported change to legislation. Two of the four research projects described how they achieved these impacts: through interaction with a policy maker (n = 1); dissemination of findings in a journal article or at a conference (n = 1); or in a report or discussion paper (n = 1). Note that one project described two pathways to achieving impact. Target audiences for the reported results included journalists (n = 3), ministers (n = 1), policy makers (n = 1), hospital or health service administrators (n = 1), and clinicians (n = 1).

Figure 1.     How policy or practice impact was achieved (click on figure to enlarge)

Figure 2. Did researchers achieve the type of impact they planned^a? (click on figure to enlarge)

^a Some projects achieved more than one type of impact

Discussion

To our knowledge, this is the first study in Australia that explores the type of impact planned and achieved in research projects from a large-scale prospective cohort study, the 45 and Up Study. The largest cohort study in the Southern Hemisphere, the 45 and Up Study, is accessed by over 660 researchers. Given the increasing interest in Australia and internationally to identify, define, and measure research impact, it is timely to describe the type of impact intended and achieved by researchers using the 45 and Up Study data.

While our study reflects a strong interest among researchers in having a policy or practice impact, few researchers nominated how they would achieve this. A limited number of studies have directly examined the impact of research on policy and practice^20-22. Some research organisations and national funding bodies are developing schemas to define and quantify research impact to expand the focus on academic outputs only. Despite the growing expectation, these initiatives are yet to impact researchers’ practice in real-world settings.^20-22 Boulding et al. found that public health research contributes substantially to impact beyond academia.²² Pathways to impact include contributing to debates on what constitutes appropriate evidence for national policy change, building relationships with stakeholders across a range of health and non-health sectors and collaborating with local authorities. Newson et al.²⁰ found that policy and practice impacts were related to significant intervention effects, the nature of the intervention and the publication of results. Whereas Read et al. found that professional networks were the most effective means of research dissemination.²¹ Also of note is the infrequent mention of interactive strategies to facilitate the translation of research findings into practice, although the concept is not new. An examination of researchers’ needs and a better understanding of the obstacles they experience in this regard would be beneficial in designing strategies to address these needs, including actively engaging with policymakers and practitioners.

The results indicate that researchers are interested in using the 45 and Up Study data to achieve impact. Indeed our study indicates that the majority of research projects in our sample intended to achieve policy or practice impact (n = 22, 88%). By the end of the reporting period, some of those projects had reported a policy or practice impact (n = 4/22, 18%). However, this analysis has shown that their capabilities to deliver impact could be improved.

There are several reasons for this difference between intention and achievement of impact. Firstly, the reporting period used in this study to assess impact, ends at the completion of the respective projects, while achieving impact is more likely to occur well after the completion of a study²⁰. Secondly, without validation of the stated impacts on policy and practice, it is unclear whether the coding schedule captured all the impact achieved within the reporting period; reliance on self-reporting here is not ideal. Lastly, although the a priori codes were developed to reflect Cohen and colleagues’ (2015) framework¹⁵, the 45 and Up Study report templates did not necessarily concur with the same definitions, creating the potential for a theoretical mismatch.

The methodological limitations in this paper were that the research project application, progress and final reports were completed for the data custodian and therefore, authors may not list a lot of detail on the impact planned and achieved. Further, a researcher’s understanding of impact may differ from that of policy makers.

There indeed appears to be little in the way of dissemination of information from the research project findings. This is very likely an artefact of timing, methodology and the nature of a routine report to a data custodian. As these are required, not all information on planned or achieved impact may be included in sufficient detail. A researcher’s understanding of impact may be different from that of a policy maker; hence, the impact planned and achieved may not be reported. For projects that did not include a final report, we considered the last progress report as the final report, and it is possible that the impact achieved was not included in these progress reports. Finally, achieving impact takes time and most likely occurs well after the project has been completed, hence not being captured in our study.

Despite these limitations, the results of this study seem to indicate that while researchers intended to have an impact through policy-oriented and academic pathways, research practice is still skewed towards traditional intellectual outputs among the 45 and Up Study research projects examined. While 48% of the sample (n = 12) reported publishing at least one journal article, few reported producing policy-oriented publications (n = 3, 12%) or engaging with the media about their research (n = 7, 28%). The literature reports several barriers to engaging in these policy-oriented activities, including a lack of organisational support and individual capacity.⁸

This study indicates a disconnect between the strategies researchers employ to inform the targeted end-users about the knowledge produced to assist with their decision-making process. It also demonstrates the need for further investigation into researchers’ processes of planning for impact and the facilitators and barriers they encounter. Therefore, a follow-up study currently underway consists of semi-structured qualitative interviews with the lead researchers to better identify barriers and facilitators to research impact, to use those insights to promote policy better and practice impact for those using the 45 and Up Study dataset.

It appears that researchers would benefit from forward impact planning. One strategy to improve impact is to have a more targeted impact plan considering the audience’s type and preferred method of accessing evidence. Planning for research impact could consider proactive approaches, engaging with policy makers and practitioners and identifying their preferred ways of accessing evidence. Planning could also redress an overemphasis on academic outputs and enable the delivery of impacts in policy and practice.

Conclusion

The majority of researchers in our study intended their research project to have policy and practice impact. While some researchers intended to reach policy and practice audiences, most did not seem to consider the needs or preferences of their knowledge users and had no established strategies to engage with them. The lack of a targeted approach to knowledge mobilisation is one of the most common barriers identified in the literature to achieving real-world impacts.

These findings suggest that 45 and Up Study researchers, and potentially researchers more broadly, may benefit from a more targeted approach to impact planning. There is a discordance between ‘intention to have impact’ and ‘achieving impact’ and further studies are required to elucidate why this is the case.

Acknowledgements

We would like to thank Emeritus Professor Bruce K Armstrong for his support and insights in developing this project.

The 45 and Up Study is managed by the Sax Institute in collaboration with major partner Cancer Council NSW; and partners: the Heart Foundation; NSW Ministry of Health; NSW Department of Communities and Justice; and Australian Red Cross Lifeblood; author MM is Chief Investigator of the 45 and Up Study.

Peer review and provenance

Externally peer-reviewed, not commissioned.

Copyright:

© 2022 Ha et al. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.