Objectives: This study tested the utility of the Intervention Scalability Assessment Tool (ISAT) with real-world interventions considered for scale-up in the Australian context. We explored user perceptions of the usefulness of the ISAT in making scalability decisions and conducted an analysis of completed ISATs.
Type of program/service: The ISAT was designed to assist policy makers/practitioners to make structured and reflective assessments of the potential scalability of their interventions.
Methods: Participants (n = 26) were invited via email and needed to be considering a population intervention for scale-up, to be able to pilot the ISAT, and to be willing to complete an online survey to provide feedback. Survey data and completed ISATs were collated and common responses were identified, with exceptions noted.
Results: Thirteen participants accepted the invitation to pilot the ISAT; six completed ISATs and returned surveys and five of each were included in the final analysis. The process by which participants completed the ISAT varied, some involving up to six stakeholders in the process and others only one. The time taken to complete the ISAT ranged from 2 hours to 3 days. Interventions considered for scalability were public health interventions and all but one involved a potential statewide scale-up. In terms of outcome, scalability scores across interventions varied, however, all but one intervention was recommended for scale-up. The remaining intervention was deemed to be promising but further information was required before assessing scalability.
Lessons learnt: The ISAT was perceived as a useful tool to assess the scalability of real-world health programs, with participants noting only minor limitations and challenges. The ISAT can be applied to a variety of population health interventions and identifies strengths and weaknesses in the evidence base for scale-up. This study revealed that decisions to scale up are often made despite potential weaknesses highlighted through the ISAT. This is likely a reflection of the critically influential role of contextual factors in decisions to scale up, whereby decision makers may balance a variety of considerations, of which evidence of scalability is only one factor.
Practical support for policy makers to guide decisions regarding the scale-up of efficacious health programs to reach larger populations is lacking.1 Although there are frameworks that guide the process of scaling up health interventions2-5, there is a dearth of practical guidance to inform decision making on the scalability of interventions6, a recognised first step when considering scaling up programs to the population level. The Intervention Scalability Assessment Tool (ISAT) is designed to address this gap, by providing policy makers and practitioners with a decision-support tool for determining suitability of health interventions for scale-up.7
The ISAT was developed by the study authors and colleagues through a series of consultations with Australian policy makers and researchers, as reported elsewhere.7 It is divided into three parts, with Parts A and B containing five domains covering aspects of the scale-up context and proposed implementation requirements (Table 1). The final Part C is a summative assessment on the scoring of questions from Parts A and B to generate a radar plot against which readiness for scale-up can be assessed, and to prompt a final recommendation regarding the suitability of the intervention for scale-up. No total score is derived overall across domains, which is deliberate because it is anticipated that the relative weight those completing the ISAT place on different domains in the decision-making processes will vary across contexts and circumstances. For example, some decision makers may be more accepting of low scores in certain domains depending on that domain’s perceived importance in the context of scaling up the specific intervention.
Table 1. Intervention Scalability Assessment Tool (ISAT) domains and objectives7
|Domain||Objective of the domain|
|A1: The problem||Consideration of the problem that is being addressed. The questions in this domain seek a description of the problem, who it affects, what it affects and how it is currently being addressed (if at all).|
|A2: The intervention||Description of the proposed program/intervention to address the problem.|
|A3: Strategic/political context||Consideration of the current strategic/political/environmental contextual factors that are potentially important influences on any intervention to be scaled up.|
|A4: Evidence of effectiveness||Consideration of the level of evidence available to support the scale-up of the proposed intervention, such as scientific literature and/or other known evaluations of the intervention.|
|A5: Intervention costs and benefits||Consideration of the known costs of the intervention delivery as well as any quantifiable benefits. This includes the results of any types of economic evaluation studies.|
|B1: Fidelity and adaptation||Consideration of whether there are any proposed changes to the intervention required for scale-up.|
|B2: Reach and acceptability||Consideration of the reach and acceptability of the intervention for the target population.|
|B3: Delivery setting and workforce||Consideration of the setting within which the intervention is delivered as well as the delivery workforce.|
|B4: Implementation infrastructure||Consideration of the potential implementation infrastructure required for scale-up.|
|B5: Sustainability||Consideration of the potential longer-term outcomes of the scale-up and how, once scaled up, the intervention could become sustainable over the medium to longer term.|
The objectives of this study were to explore user experiences in completing the ISAT and the perceived usefulness of ISAT in making scalability decisions.
Participants were recruited via an email, which was sent to a range of Australian health policy makers, implementation scientists and practitioners with experience in scaling up health interventions, who had been consulted previously during the development of the ISAT (n = 18). Further detail on these participants have been reported elsewhere.7 Invited participants nominated further suitable participants (n = 8) via a process of passive snowball recruitment, resulting in a total of 26 participants being invited. Participants needed to be:
A copy of the ISAT and a link to an online survey were sent to participants in the period January–February 2019. The survey included closed and open-ended questions seeking perceptions on the process of ISAT completion, appropriateness of the content and the perceived usefulness of the ISAT for scale-up decision making.
Due to the small number of surveys, the closed-ended questions were analysed descriptively. The interview data was analysed by the lead author (KL) along with the responses to the open-ended survey questions from which common responses/concepts on the utility of the ISAT were identified. The interpretation of the open-ended responses and interview data were validated and further refined with another co-author (AG) and any differences between KL and AG were clarified by discussion. Observations regarding the content of completed ISATs were also collated and discussed between authors (KL, AB and AG). Ethics approval was obtained from the University of Sydney’s Human Research Ethics Committee (2017/828).
Thirteen invitees accepted the invitation to pilot the ISAT. The remaining invitees did not respond (n = 5) or did not meet the criteria for participation (n = 8). Six participants returned completed ISATs and surveys. One ISAT and one survey from different participants and interventions had key information missing and were therefore excluded, leaving five ISATs and five surveys in the final analysis. Another participant was unable to complete the ISAT because efficacy testing of their intervention was ongoing and scalability could not be assessed. The participant was, however, keen to provide their views on the perceived utility of the ISAT given their experience in scale-up and this feedback was obtained via a short telephone interview conducted by KL, during which notes were taken for further analysis. The remaining six participants who had agreed to take part did not respond with completed ISATs or surveys despite being reminded.
Of the six participants whose survey and interview feedback were used in the ISAT pilot analysis; four were from state/local health organisations, one was from a nongovernment organisation and one was from an academic institution. Four participants were from New South Wales, one was from Victoria and one was from Tasmania. Participants were either the program manager of the intervention being considered for scale-up and/or an integral part of the scale-up decision-making process.
As it is recommended that the ISAT be completed collaboratively with a range of stakeholders, three out of five participants reported completing the ISAT collaboratively, involving two to six stakeholders in the process. These stakeholders were reported to be academics, representatives from funding agencies and service providers. The remaining two ISATs were completed by the program managers for the interventions being considered for scale-up.
Three out of five participants strongly agreed/agreed that completing the ISAT was a straightforward process. The length of time taken to complete the ISAT ranged from 2 hours to 3 days in total. Those who took less time to complete the ISAT also had fewer stakeholders participating in the process. Despite this variation, participants felt that the time taken to complete the ISAT was worthwhile as they reported they could see how that information used to complete the ISAT could be useful for others and that the questions asked in the ISAT were relevant to the scalability assessment of their intervention. All participants strongly agreed/agreed that completing the ISAT was a useful exercise and it provided them a structured process for making the decision on the scalability of the intervention. Three out of five indicated that the ISAT highlighted that their selected intervention likely required modifications to enhance scalability, prompting further consultation with colleagues.
One of the underlying challenges reported by participants was in applying some of the generic questions in the ISAT to their specific intervention, for example, some of the questions did not apply in their context. Other limitations reported were that the ISAT was less applicable for interventions with an emerging or unclear evidence base and that it largely assumes a research-driven process of scaling up, not easily accommodating interventions driven by practice.8
A range of public health interventions were covered in the piloting of the ISAT (Table 2). All but one were considering statewide scale-up prior to completing the ISAT. Intervention B had approval to scale up and had commenced the scale-up process, while others were yet to determine suitability for scale-up.
Table 2. Interventions considered for scale-up
|Intervention||Intervention type||Lead agency||Proposed level of scale-up||Status before starting ISAT||Final recommendation|
|A||Strategies and support for sporting clubs to improve sun protection knowledge, practice and behaviour among men >40 years||Nongovernment organisation partnership with state government||Multiple sites within a region||Being considered for scale-up||Merits scale-up|
|B||Community-based program engaging family members in positive lifestyle role modelling and effective parenting strategies to improve a range of physical activity behaviours||State government and university||Statewide scale-up||Decision to scale up already made. In early stages of scaling up||Merits scale-up|
|C||Training for clinicians to deliver constraint-induced movement therapy for stroke rehabilitation||Local Health District||Statewide scale-up||Being considered for scale-up||Merits scale-up|
|D||Sexual health promotion youth services including resources, training and mentoring for youth workers||State government||Statewide scale-up||Being considered for scale-up||Merits scale-up|
|E||Nutrition social marketing campaign to influence healthy eating and connecting the local food industry with the community||State government||Statewide scale-up||Being considered for scale-up||Promising, but further information/ planning is warranted|
The level of detail provided across and within each of the completed ISATs varied considerably. Four out of five scored their intervention highly for the ‘Problem’, ‘Intervention’ and the ‘Political/Strategic Context’ domains (in Part A) (see Figure 1). Future Implementation Planning (Part B) domains scored lower overall, with greater variation across the interventions, indicating that implementation aspects of interventions were generally deemed less favourable for scaling up compared with the domains in Part A.
Figure 1. Results of the Intervention Scalability Assessment Tool (ISAT) radar plots for all interventions (click to enlarge)
Despite wide-ranging final scores, the final recommendation made for the interventions in this study, apart from Intervention E, was to scale up. Although it had high scores across some domains, Intervention E was deemed a promising intervention requiring further modifications, as the program manager intended to consult additional stakeholders before making a final decision on scale-up. Interventions A, C and D scored highly across the first three domains in Part A, but had varying results in Part B.
Low to moderate scores on the costs domain (domain A5) were common across all interventions. Only two of the five interventions had some form of economic analysis conducted for their intervention, while for the others, there was basic or no information on costs available.
According to our pilot, the ISAT facilitates assessment and decision making on the scalability of health interventions and has potential for use in real-world settings. Participants were positive about the ISAT and despite variation in how it was used, felt it contributed to their decision making. Our findings suggest that evidence for scalability may not be the only factor determining final decisions to scale up an intervention, as demonstrated by the variability in scores leading to similar recommendations that the intervention merited scale-up.
Survey feedback from participants indicated that the ISAT was completed in a flexible manner, modified to suit the individual context. Although a team approach was recommended, our study showed that the ISAT may be completed by fewer people for the sake of expediency. The danger in doing so however, is not obtaining a variety of perspectives which is important as part of scalability assessments.9
One challenge noted by the study participants was that there was often insufficient information on intervention costs or economic analyses to make an informed assessment. This highlights a key issue in scalability decision making; that often the long-term sustainability of interventions is dependent on economic appraisals of interventions10,11, yet this information is often not available at the time of the decision-making process.1,12 A strength of the ISAT is that such gaps may be identified and addressed if they are judged to be critical to decision making. For example, in the case of Intervention E, the policy team decided to defer a decision on scale-up until additional stakeholders were consulted.
This pilot has highlighted the importance of context in decision making on intervention scalability. We found that despite the ISAT exposing weaknesses in some domains, decision makers still considered scaling up interventions for other reasons.13 For example, higher scores in the ‘problem’ and/or ‘political/strategic context’ domains might influence the final decision to scale up, even where the evidence of scalability potential in other domains is weak. Nonetheless, use of the ISAT lends transparency to scale-up decisions, through identifying future risks and consequently, areas requiring additional planning for scaling up. This pilot has provided support for some of the key factors in the literature that are considered critical for scalability such as evidence of efficacy and implementation planning1,2, as well as identified the tension between the political imperative to scale-up and the evidence base to support it.
The small sample size for this pilot, as well as the specific Australian context, limits the generalisability of the observations. However, the purpose was to test the utility of the ISAT prior to wider distribution, and this study has demonstrated the acceptability of the tool for the intended user group.
The ISAT provides a structured process for assessing the scalability of population-level health interventions and flags potential issues for implementation at scale. The pre-emptive identification of gaps in the evidence for scalability enables planning to mitigate potential risks if scale-up proceeds. Our study shows the ISAT can likely be used in a variety of settings and is practical, flexible and adaptable to different contexts. In the future, the tool could be further refined for use in other non-health, human service–related interventions, although further larger-scale study may be required.
Externally peer reviewed, not commissioned.
© 2020 Lee et al. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.