Value of consultation in establishing a public health research network: lessons from APPRISE December 2021, Volume 31, Issue 5

Introduction

The Australian Partnership for Preparedness Research on Infectious diseaSe Emergencies (APPRISE) is a national research collaboration that spans laboratory, clinical and public health research.¹ It is funded by the Australian National Health and Medical Research Council (NHMRC) as a Centre of Research Excellence (CRE). APPRISE works to provide the foundations for an ongoing, coordinated national research effort to effectively inform the national response to emerging infectious diseases.

APPRISE, which was funded in 2016 following a specific call for applications, had a novel requirement to undertake a consultative research priority-setting exercise in its first year prior to the full funding being released.² This new funding model was used to ensure that the research team was well engaged with the end users of the research from the outset. The consultation process required broad stakeholder engagement around shared research priorities to ensure that the resulting research is of maximum benefit to public health responders and decision makers for infectious disease emergencies.

The consultation process

The consultation was conducted over a 6-month period by an external consultancy, Nous Group, and involved stakeholder mapping, interviews with 35 key stakeholders, four workshops (in Canberra, Brisbane, Melbourne and Sydney), a consolidation workshop with APPRISE investigators and the preparation of a comprehensive report.

---

Box 1. APPRISE stakeholders

Academic (researchers, centres and networks)

• APPRISE investigators (public health, clinical and laboratory researchers)
• Infectious disease researchers – national and international
• Infectious disease research institutes and Centres of Research Excellence
• National and international research networks

Government (agencies, networks and regulators)

• National Health and Medical Research Council
• Australian Health Protection Principal Committee
• Communicable Diseases Network Australia
• Australian Government Department of Health Office of Health Protection
• State government departments of health

Public health laboratories

• Clinical and professional (medical/emergency management and animal health)
• Medical colleges and representative bodies
• Animal health stakeholders including laboratories, veterinarians and researchers

Representative groups

• Aboriginal Community Controlled Health Organisations
• First nations community representative groups and research organisations

Industry

• Pathology laboratories
• Pharmaceutical and technology companies

---

Given the context and the unique set-up involving significant financial and time investments, we sought to evaluate the impact of the consultation requirement on the APPRISE research team and key stakeholders who participated. We surmised that potential impacts of the consultation could include changes to the agreed research priorities, the level of engagement and motivation of participants and stakeholders, identification of gaps in the network, and the formation of a unified team addressing a common purpose.

Exploring the impact of the consultation has potential value for the NHMRC and APPRISE researchers, as well as for external stakeholders interested in and potentially affected by APPRISE research. For APPRISE researchers, we sought to clarify success factors and address challenging issues to facilitate ongoing strong collaboration efforts and research impact. For external stakeholders, the evaluation aimed to identify ways to improve engagement and to better understand the benefits and limitations of the consultative model for the establishment of similar research networks in the future. The NHMRC and stakeholders expressed genuine interest in learning from an evaluation of the set-up and initiation of APPRISE.

APPRISE is a research collaboration spanning laboratory, clinical and public health research. Seeking to understand how the consultation process impacted the establishment of the research team and its early progress has broader implications for network building in public health. Lessons can also be learnt about the impact of consultation on a collaborative research program.

Methodology

The evaluation was performed as a modified Delphi technique in two stages from May to September 2018. The Delphi technique is a group facilitation strategy that helps to establish group consensus.³ It functions as a virtual focus group, providing the ability to identify and rank relevant issues through multiple rounds of questions, without requiring a physical meeting.

The first round of questions (stage one) was delivered via a mix of one-on-one interviews and an online survey. Twenty-three individual invitations were sent to people closely involved in APPRISE (n = 19) and the consultation (n = 4) (including 13 invitations for interview and 10 for the survey). The individuals defined as ‘closely involved’ included investigators, reference group stakeholders, representatives from the NHMRC and the external consultants (Nous Group). Closely involved individuals were those actively involved in the set-up of the APPRISE grant call, the grant application and the set-up of the consultation process. A generic email invitation to complete the survey was sent to individuals who were less closely involved in the consultation (n = 155). These individuals had been invited to participate in the consultation process once it was established. In total, 178 individuals (23 closely involved and 155 less closely involved) were approached to participate in stage one of the study. Study data were collected and managed using REDCap electronic data capture tools hosted at the University of Melbourne.⁴

The second round of questions (stage two) was delivered exclusively via a REDCap online survey. All individuals who were invited to participate in stage one of the study – including both those who participated in stage one and those who didn’t take part – were invited to participate in stage two (n = 178) via individual and generic email invitations.

The stage one and two evaluation surveys were structured around three periods in the APPRISE grant as described in Table 1. These three periods encompass the time leading up to the original grant submission (pre-consultation period); from the grant approval through the externally conducted consultation to the submission of the consultation report (consultation period); and then after the confirmation of ongoing funding (post-consultation period).

 

Table 1.     APPRISE grant periods and activities as defined for evaluation purposes

Period	Pre-consultation	Consultation	Post-consultation
Timeframe	Before July 2016	July 2016–June 2017	After October 2017
Activities	Call for submissions Grant writing Grant submission to NHMRC	Grant award Define terms of reference for consultation Seek expressions of interest for consultants Appoint consultants Conduct consultation Consultation report writing and submission to NHMRC	Feedback on consultation report from NHMRC and AHPPC Confirmation of ongoing funding Start of research activities

NHMRC = National Health and Medical Research Council; AHPPC = Australian Health Protection Principal Committee

 

The stage one survey considered the strengths and challenges of each period by asking broad questions about processes and discussions that were useful or unhelpful at each point in time. To gain a more nuanced understanding of participants’ inputs, we asked about their level of involvement in the consultation process, their overall satisfaction with the consultation, and what they considered to be the strengths and limitations of the consultation requirement.

Responses to each stage one question were compiled and thematically analysed.⁵ The analysis involved coding (or categorising) each comment or response and then drawing all of the coded responses into groupings. The codes and thematic groupings were discussed and agreed by both study investigators (MS and PM).

The stage two survey asked respondents to rank the themes identified in stage one across the three periods according to how useful/influential/helpful/constructive or unhelpful/frustrating/challenging they found the issues. Ranking questions were evaluated by calculating a weighted score, where first preference scored 3 points, second preference scored 2 points and third preference scored 1 point.

Ethics approval for the study was provided by the University of Melbourne (1852021.1).

Results

Evaluation participants

Twenty-one people participated in stage one of the study (Figure 1). Our participation rate was 74% of invitees closely involved in APPRISE (17/23), and only 2.5% of those less closely involved (4/155). Twelve people took part in a telephone or face-to-face interview, and the remaining nine participated in the online survey. All participants were asked the same questions. Seventeen participants (81%) from stage one participated in stage two, 14 of whom were closely involved and three less closely involved in the consultation (Figure 1).

Figure 1.     Participant recruitment to stage one and stage two of the study (click to enlarge)

Most participants were APPRISE investigators, with the remaining participants representing members of the APPRISE Expert Reference Group or other Working Group, the NHMRC, the external consultants (Nous Group) and APPRISE collaborators.

APPRISE set-up and the pre-consultation period

Most participants (75%) became involved in APPRISE before the submission of the NHMRC grant application. More than 20 useful and influential factors were identified that influenced the initial stages of APPRISE. These were grouped into five themes. The most influential factor affecting the proposal was considered to be the national and international infectious disease context (including the aftermath of the 2009 H1N1 influenza pandemic⁶, the threat of H5N1 avian influenza⁷ and the West African Ebola outbreak⁸). The second most influential factor was how the diverse research and policy interests were drawn together into a coherent structure that complemented existing bodies and structures while simultaneously promoting a national research network with a strong equity focus. The other three themes (in order of influence) were: 1) the high calibre of the involved researchers and the use of strategic capacity seeking; 2) communication between investigators; and 3) and the co-ordination of the proposal development.

Participants identified 12 unhelpful or frustrating factors in the pre-consultation period. These factors grouped into three categories: 1) communication challenges; 2) personal challenges for engagement; and 3) challenges with balancing competing perspectives and priorities within a broad and, at times, unclear scope. Participants ranked these three categories similarly, with no clear hierarchy.

The consultation period

Most participants (17/21, 81%) were actively involved in the consultation process, participating in one or more interviews and workshops. Twelve different issues were identified as constructive in stage one, which grouped into four themes (Table 2). The most significant of these, when ranked in stage two of the study, was the contribution to network development. This included the benefit of communicating about APPRISE and its aims to stakeholders, enhanced collaboration and networking (particularly with key government stakeholders such as the Communicable Diseases Network Australia) and the opportunity to engage with reluctant or unhappy stakeholders.

Three themes developed when we asked participants to identify the biggest challenges with the consultation. Defining and managing the expectations of a broad range of stakeholders was considered the biggest challenge. The different, and at times competing, expectations of research stakeholders and public health practitioners was highlighted. Contributing to this was a frequent lack of good understanding of the different roles of these stakeholders. For example, the priorities of researchers to gain detailed understanding of an infectious disease situation differed from the public health imperative to respond quickly and with often limited resources. The other challenging themes were balancing the need for consultation with research momentum and enabling the involvement of diverse participants – particularly Aboriginal and Torres Strait Islander organisations and private sector (industry) stakeholders.

When asked to identify the most important impacts and outcomes of the consultation, participants described more than 20 different issues. These grouped into six themes (Table 2) and, overwhelmingly, the most important of these was the consolidation of APPRISE as a national research entity. The consolidation included developing relationships within a complex stakeholder network, particularly with the Australian Government Department of Health and its committees and with other infectious-disease related Centres of Research Excellence.

The post-consultation period and research phase

Questions regarding the post-consultation period focused on research collaboration. Participants identified more than 15 processes that facilitate collaboration, which grouped into five themes (Table 2). The most helpful processes were those that assisted productive involvement, including dedicated project staff, clearly defined research priorities and encouraged collaborative thinking. Engagement with key communities and stakeholders – including Aboriginal Community Controlled Health Organisations and government committees such as the Communicable Diseases Network Australia – through activities such as the annual meeting and development of an early career researcher group was the next highest ranked theme.

Participants identified 10 factors that were unhelpful or frustrating to collaborative research efforts. These factors grouped into three themes. Two themes ranked almost equally – the impact of the broad scope of APPRISE resulting in potentially unrealistic expectations, and challenges in drawing together a diverse research team across different disciplines. The third theme identified challenges associated with communication across a large and geographically distributed team, and a large number of ongoing discussions were also identified as unhelpful factors, although they were not ranked as highly.

 

Table 2.     Themed key findings from the APPRISE consultation evaluation process, May–September 2018

Evaluation issue	Themed factors identified	Weighted Scorea
Most constructive parts of the consultation process	Network development and engagement	38
	Impact on research and translational strategies	33
	Well-managed process	23
	Encouraged personal committment	5
Biggest challenges of the consultation	Defining and managing the expectations of a broad range of stakeholders	49
	Balancing the need for consultation with research momentum	27
	Enabling the involvement of diverse participants	26
Factors facilitating collaboration	Support staff and strategic thinking	24
	Engagement with key communities and stakeholders	12
	Communications within network	10
	Calibre and enthusiasm of research team	8
	Funding calls	6
Factors that were unhelpful or frustrating to collaborative research efforts	Challenges in drawing together a diverse research team across different disciplines	24
	The impact of the broad scope of APPRISE resulting in potentially unrealistic expectations	22
	Communication challenges	13
Impacts and outcomes of the consultation	Consolidation of APPRISE as a national research entity	40
	Strengthening the research team	16
	Highlighting the need for long-term thinking	16
	Documenting the consultation in a report	14
	Uncertainty and delay during the consultation review	12
	Appreciation of the time invested	3

^a Weighted scores were calculated by allocating 3 points where a theme was ranked first preference, 2 points for second preference and 1 point for third preference. The total weighted score reflects responses from all participants.

 

Finally, participants were asked to identify the most significant strengths and limitations of the consultation approach (Figure 2). Ten strengths and 11 limitations were identified in stage one. Participants selected the three most significant strengths and limitations in stage two of the study. The most important strengths of the consultation approach centred on the establishment of APPRISE as an entity in the preparedness field with its networking opportunities, its benefit for policy-focused research and principles for ongoing communications (Figure 2). The most important limitations were the need to manage expectations, the unfinished nature of the consultation process which requires ongoing involvement, the limited time and energy of investigators to action the proposals, the possibility that broad research priorities could lead to few defined outcomes, and the fact that the consultation process caused a delay to the start of research projects.

Figure 2.     Strengths and limitations of the consultation approach (click to enlarge)

Discussion

Important lessons can be learnt from the establishment and consultation process for APPRISE. The call for a proposal to address infectious disease research preparedness resulted from extensive discussions between the Australian Government Department of Health, the NHMRC and researchers following the 2009 H1N1 influenza pandemic.⁶ A sense of relevance and urgency was created by the contemporary Ebola⁸ and H5N1 and H7N9 influenza^7-9 outbreaks. Providing further impetus, European preparedness networks were being established^10,11, along with the development of the Australian National Framework for Communicable Disease Control.¹² Unsurprisingly, the national and international infectious diseases context was found to be a major influencer of motivation for and engagement with the APPRISE set-up. The coronavirus disease 2019 (COVID-19) pandemic¹³ emerged after the completion of this study, throwing into sharp relief the relevance and importance of pre-established research networks and relationships.

As a multidisciplinary research undertaking, the consolidation of a national research network was a key consultation outcome. Network development has multiple meanings. In this context we are referring to the strengthening of communication and development of deeper collaborative relationships between stakeholders rather than the technical (and quantifiable) definition associated with social network analysis. This type of cross-sectoral engagement is considered essential for addressing the complex issues in health protection and preparedness research^14-16, and was specifically referenced in the evaluation responses. Although network consolidation and stakeholder engagement were constructive and are essential to facilitating research preparedness¹⁷, these factors also generated broad expectations which need ongoing management.

The use of a modified Dephi technique enabled a more detailed understanding of stakeholder perspectives and priorities than a simple interview or survey.¹⁸ Many issues that were identified in stage one of the study by multiple respondents, including the delay in starting research and the importance of funding, did not rank among the most prominent issues in stage two. The anonymity afforded by online engagement enabled a degree of openness that may not have been possible in a face-to-face setting.¹⁸ Robust retention of participants between stage one and stage two limited one potential shortcoming of the Delphi process, where attrition between stages hinders analysis.¹⁸

Limitations

The participants in the evaluation were predominantly APPRISE investigators and key stakeholders closely involved in APPRISE. This is a notable limitation in the breadth of opinions gathered, with the results reflecting the opinions of the stakeholders most heavily engaged in and affected by the APPRISE set-up and consultation and not those less closely involved. There are many possible reasons for the poor response from participants who were less closely involved, including the delay between the consultation (in early 2017) and the evaluation process (in mid-2018), the online nature of the survey¹⁹, the salience of the evaluation to these individuals²⁰, and the lack of personalised invitation. Non-response bias has been studied in the context of health-related research studies themselves^21-23 but to our knowledge, not in the context of evaluations. No demographic data was collected to enable comparison of either respondents and non-respondents or those who were closely involved and those less closely involved. More involvement in the evaluation from those less closely involved in APPRISE would have provided a broader understanding of the impact of the consultation process. The value of the results is that they reflect the experiences and opinions of a close group who devised and guided the consultation process, providing a useful insider perspective.

Broader issues for public health research

The consultation process for APPRISE has highlighted some vital broader issues for public health research. It has demonstrated the importance of network development, particularly across multiple research domains and with a complex network of external stakeholders. Wide consultation to enable ongoing funding has facilitated open discussion about strategic priorities for infectious disease preparedness, strengthening engagement and a more research-ready public health emergency network, although this consultation did not change the issues that were identified as research priorities. Strategic and long-term thinking, plus a focus on research translation were identified as useful and constructive parts of the unique consultation. APPRISE has successfully brought together researchers from diverse specialties and enabled interaction with policy makers – some for the first time. A collaborative approach can help to build the relationships necessary to identify and address research priorities for effective responses to infectious disease emergencies. An additional benefit of having a pre-established network is that it provides a rapid way to disseminate research funding in the event of an emergency.¹⁷ The emergence of  COVID-19 within the APPRISE grant period tested this capacity with APPRISE funds allocated to urgent COVID-19 research before the declaration of a worldwide pandemic on 11 March 2020.^13. Through special funding provisions with the NHMRC, additional funds were provided to support additional COVID-19 research, including a national serosurvey, interventional and observational clinical trials, and tools for limiting the spread of COVID-19 in aged care facilities.

Establishing APPRISE as a preparedness network was the beginning of a broader and essential effort to integrate research into infectious disease emergency responses. The Australian health system is complex, with responsibilities for health spread between state and federal governments. Aside from establishing the mechanisms for a research response to infectious disease emergencies, APPRISE also needs to continue to demonstrate how research findings feed into existing decision-makingl hierarchies. An ideal system integrating research and response should include multidisciplinary collaborations with prioritised, well-funded and sustainable research activities.¹⁴ Now that APPRISE has established a national research network, the investment of time, effort and expertise needs to be drawn on to make lasting changes in how research is integrated into public health emergency decision making. Working with the imprecise boundary between routine public health surveillance and research in the event of a public health emergency is an important aspect of the work. Defining how research is written into infectious disease emergency response plans will be an important measure of success.

This evaluation explores in detail the insider perspective of the impact of consultation on the establishment of a cross-disciplinary and cross-sectoral research entity with a translational public health imperative. The initial consultation process to enable the release of ongoing funding was an integral part of establishing APPRISE as a cohesive research network in Australia. Managing the relationships and diverse expectations developed during the consultation period is an ongoing task for APPRISE. While important progress has been made, the COVID-19 pandemic has demonstrated that there is still much work to be done to integrate research findings and decision making in public health emergencies. These are important challenges, and ones that we must confront to better prepare for future infectious disease emergencies. Consultation was a challenging but necessary and valuable component of the set-up of this novel complex public health research network.

Conclusion

This evaluation has clarified the strengths and limitations of a consultation process in establishing APPRISE as a preparedness research entity in Australia. The outcomes and conclusions of the evaluation comprise the reflections of those most closely involved in APPRISE and the consultation process. The consultation has helped to establish a critical research network while highlighting the ongoing need for stakeholder engagement and expectation management. Understanding the full range of consequences of broad consultation is informative for further network building in public health.

Acknowledgements

This study was funded by a National Health and Medical Research Council Grant (APP1116530).

Peer review and provenance

Externally peer reviewed, not commissioned.

Copyright:

© 2021 Smith and Massey. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.