The Intervention Scalability Assessment Tool: a pilot study assessing five interventions for scalability June 2020, Volume 30, Issue 2

Introduction

Practical support for policy makers to guide decisions regarding the scale-up of efficacious health programs to reach larger populations is lacking.¹ Although there are frameworks that guide the process of scaling up health interventions^2-5, there is a dearth of practical guidance to inform decision making on the scalability of interventions⁶, a recognised first step when considering scaling up programs to the population level. The Intervention Scalability Assessment Tool (ISAT) is designed to address this gap, by providing policy makers and practitioners with a decision-support tool for determining suitability of health interventions for scale-up.⁷

The ISAT was developed by the study authors and colleagues through a series of consultations with Australian policy makers and researchers, as reported elsewhere.⁷ It is divided into three parts, with Parts A and B containing five domains covering aspects of the scale-up context and proposed implementation requirements (Table 1). The final Part C is a summative assessment on the scoring of questions from Parts A and B to generate a radar plot against which readiness for scale-up can be assessed, and to prompt a final recommendation regarding the suitability of the intervention for scale-up. No total score is derived overall across domains, which is deliberate because it is anticipated that the relative weight those completing the ISAT place on different domains in the decision-making processes will vary across contexts and circumstances. For example, some decision makers may be more accepting of low scores in certain domains depending on that domain’s perceived importance in the context of scaling up the specific intervention.

Table 1.     Intervention Scalability Assessment Tool (ISAT) domains and objectives⁷

The objectives of this study were to explore user experiences in completing the ISAT and the perceived usefulness of ISAT in making scalability decisions.

Methods

Participants were recruited via an email, which was sent to a range of Australian health policy makers, implementation scientists and practitioners with experience in scaling up health interventions, who had been consulted previously during the development of the ISAT (n = 18). Further detail on these participants have been reported elsewhere.⁷ Invited participants nominated further suitable participants (n = 8) via a process of passive snowball recruitment, resulting in a total of 26 participants being invited. Participants needed to be:

• Considering an intervention for scale-up
• Able to pilot and complete the ISAT
• Willing to provide feedback on their experience.

A copy of the ISAT and a link to an online survey were sent to participants in the period January–February 2019. The survey included closed and open-ended questions seeking perceptions on the process of ISAT completion, appropriateness of the content and the perceived usefulness of the ISAT for scale-up decision making.

Due to the small number of surveys, the closed-ended questions were analysed descriptively. The interview data was analysed by the lead author (KL) along with the responses to the open-ended survey questions from which common responses/concepts on the utility of the ISAT were identified. The interpretation of the open-ended responses and interview data were validated and further refined with another co-author (AG) and any differences between KL and AG were clarified by discussion. Observations regarding the content of completed ISATs were also collated and discussed between authors (KL, AB and AG). Ethics approval was obtained from the University of Sydney’s Human Research Ethics Committee (2017/828).

Results

Participant characteristics

Thirteen invitees accepted the invitation to pilot the ISAT. The remaining invitees did not respond (n = 5) or did not meet the criteria for participation (n = 8). Six participants returned completed ISATs and surveys. One ISAT and one survey from different participants and interventions had key information missing and were therefore excluded, leaving five ISATs and five surveys in the final analysis. Another participant was unable to complete the ISAT because efficacy testing of their intervention was ongoing and scalability could not be assessed. The participant was, however, keen to provide their views on the perceived utility of the ISAT given their experience in scale-up and this feedback was obtained via a short telephone interview conducted by KL, during which notes were taken for further analysis. The remaining six participants who had agreed to take part did not respond with completed ISATs or surveys despite being reminded.

Of the six participants whose survey and interview feedback were used in the ISAT pilot analysis; four were from state/local health organisations, one was from a nongovernment organisation and one was from an academic institution. Four participants were from New South Wales, one was from Victoria and one was from Tasmania. Participants were either the program manager of the intervention being considered for scale-up and/or an integral part of the scale-up decision-making process.

Online survey responses

As it is recommended that the ISAT be completed collaboratively with a range of stakeholders, three out of five participants reported completing the ISAT collaboratively, involving two to six stakeholders in the process. These stakeholders were reported to be academics, representatives from funding agencies and service providers. The remaining two ISATs were completed by the program managers for the interventions being considered for scale-up.

Three out of five participants strongly agreed/agreed that completing the ISAT was a straightforward process. The length of time taken to complete the ISAT ranged from 2 hours to 3 days in total. Those who took less time to complete the ISAT also had fewer stakeholders participating in the process. Despite this variation, participants felt that the time taken to complete the ISAT was worthwhile as they reported they could see how that information used to complete the ISAT could be useful for others and that the questions asked in the ISAT were relevant to the scalability assessment of their intervention. All participants strongly agreed/agreed that completing the ISAT was a useful exercise and it provided them a structured process for making the decision on the scalability of the intervention. Three out of five indicated that the ISAT highlighted that their selected intervention likely required modifications to enhance scalability, prompting further consultation with colleagues.

One of the underlying challenges reported by participants was in applying some of the generic questions in the ISAT to their specific intervention, for example, some of the questions did not apply in their context. Other limitations reported were that the ISAT was less applicable for interventions with an emerging or unclear evidence base and that it largely assumes a research-driven process of scaling up, not easily accommodating interventions driven by practice.⁸

Completed and returned ISATs

A range of public health interventions were covered in the piloting of the ISAT (Table 2). All but one were considering statewide scale-up prior to completing the ISAT. Intervention B had approval to scale up and had commenced the scale-up process, while others were yet to determine suitability for scale-up.

Table 2.     Interventions considered for scale-up

The level of detail provided across and within each of the completed ISATs varied considerably. Four out of five scored their intervention highly for the ‘Problem’, ‘Intervention’ and the ‘Political/Strategic Context’ domains (in Part A) (see Figure 1). Future Implementation Planning (Part B) domains scored lower overall, with greater variation across the interventions, indicating that implementation aspects of interventions were generally deemed less favourable for scaling up compared with the domains in Part A.

Figure 1.     Results of the Intervention Scalability Assessment Tool (ISAT) radar plots for all interventions (click to enlarge)

Figure 1. Results of the Intervention Scalability Assessment Tool (ISAT) radar plots for all interventions

Despite wide-ranging final scores, the final recommendation made for the interventions in this study, apart from Intervention E, was to scale up. Although it had high scores across some domains, Intervention E was deemed a promising intervention requiring further modifications, as the program manager intended to consult additional stakeholders before making a final decision on scale-up. Interventions A, C and D scored highly across the first three domains in Part A, but had varying results in Part B.

Low to moderate scores on the costs domain (domain A5) were common across all interventions. Only two of the five interventions had some form of economic analysis conducted for their intervention, while for the others, there was basic or no information on costs available.

Discussion

According to our pilot, the ISAT facilitates assessment and decision making on the scalability of health interventions and has potential for use in real-world settings. Participants were positive about the ISAT and despite variation in how it was used, felt it contributed to their decision making. Our findings suggest that evidence for scalability may not be the only factor determining final decisions to scale up an intervention, as demonstrated by the variability in scores leading to similar recommendations that the intervention merited scale-up.

Survey feedback from participants indicated that the ISAT was completed in a flexible manner, modified to suit the individual context. Although a team approach was recommended, our study showed that the ISAT may be completed by fewer people for the sake of expediency. The danger in doing so however, is not obtaining a variety of perspectives which is important as part of scalability assessments.⁹

One challenge noted by the study participants was that there was often insufficient information on intervention costs or economic analyses to make an informed assessment. This highlights a key issue in scalability decision making; that often the long-term sustainability of interventions is dependent on economic appraisals of interventions^10,11, yet this information is often not available at the time of the decision-making process.^1,12 A strength of the ISAT is that such gaps may be identified and addressed if they are judged to be critical to decision making. For example, in the case of Intervention E, the policy team decided to defer a decision on scale-up until additional stakeholders were consulted.

This pilot has highlighted the importance of context in decision making on intervention scalability. We found that despite the ISAT exposing weaknesses in some domains, decision makers still considered scaling up interventions for other reasons.¹³ For example, higher scores in the ‘problem’ and/or ‘political/strategic context’ domains might influence the final decision to scale up, even where the evidence of scalability potential in other domains is weak. Nonetheless, use of the ISAT lends transparency to scale-up decisions, through identifying future risks and consequently, areas requiring additional planning for scaling up. This pilot has provided support for some of the key factors in the literature that are considered critical for scalability such as evidence of efficacy and implementation planning^1,2, as well as identified the tension between the political imperative to scale-up and the evidence base to support it.

Limitations of this study

The small sample size for this pilot, as well as the specific Australian context, limits the generalisability of the observations. However, the purpose was to test the utility of the ISAT prior to wider distribution, and this study has demonstrated the acceptability of the tool for the intended user group.

Conclusions

The ISAT provides a structured process for assessing the scalability of population-level health interventions and flags potential issues for implementation at scale. The pre-emptive identification of gaps in the evidence for scalability enables planning to mitigate potential risks if scale-up proceeds. Our study shows the ISAT can likely be used in a variety of settings and is practical, flexible and adaptable to different contexts. In the future, the tool could be further refined for use in other non-health, human service–related interventions, although further larger-scale study may be required.

Acknowledgements

This study was funded by The Australian Prevention Partnership Centre, which is administered by the Sax Institute.

Peer review and provenance

Externally peer reviewed, not commissioned.

Copyright:

© 2020 Lee et al. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.