Improving the quality of healthcare: a cross-sectional study of the features of successful clinical networks December 2018, Volume 28, Issue 4

Introduction

The next frontier for evidence based healthcare is to develop the science of its implementation into routine care across health systems.¹ Internationally, networks of clinical experts are considered important vehicles for this implementation, as they can provide ‘bottom up’ views on tackling complex problems and champion change at the clinical interface.^2,3 However, evidence for their success is largely anecdotal and experiential^2,3, or focused on individual clinical areas.^4-6 The science to support clinical network design has not kept pace with networks’ rapid operational development.⁷

We define ‘clinical networks’ as networks of clinicians and consumers that aim to improve clinical care and service delivery using a collegial approach to agree on and implement a range of strategies.^2,3 A small number of qualitative and comparative case studies have investigated features of clinical networks^8-11, and have suggested that strong leadership, an inclusive and collaborative culture with widespread clinician and stakeholder engagement, and adequate resources tend to be associated with success. An appropriate organisational structure has been shown to be necessary for changing processes and implementing quality improvement activities; a senior strategic leadership group, with implementation at the local level, and a focused, strategic approach to the selection of evidence based programs, has been the most successful structure.^9,10,12 External factors such as supportive policy environments, health reorganisations and financial targets also influenced outcomes.^10,12 To our knowledge, there has been no quantitative examination of the factors influencing clinical network success.

Nineteen diverse clinical networks established by the Agency for Clinical Innovation (the Agency) in New South Wales (NSW), Australia, provided a unique opportunity to quantitatively assess what features of clinical networks influence their ability to drive improvements in quality of care and facilitate system-wide change.^3,13,14 These state-funded clinical networks span multiple disciplines across a large geographical region and have a system-wide focus where clinicians identify and advocate for models of service delivery and quality improvement activities in specialty health service areas. The clinical networks work in collaboration with the NSW Ministry of Health, Local Health Districts and other associated organisations¹⁵, and operate in a similar manner to those in the UK¹⁶, parts of Europe⁸, Canada¹⁷ and the US.⁴

We present findings on whether clinical networks can be effective in producing system-wide improvement, and which factors increase their success. Given the complexity and depth of investigation of this study, a number of papers have been, or are in the process of being, published: description of the study protocol¹⁸, methods and psychometric properties^19,20; a qualitative prestudy to inform the conceptual model¹⁰; and a qualitative study to assist with interpreting quantitative results.²¹ In this paper, to test the conceptual model¹⁸, we report the main analyses for the hypotheses that successful clinical networks have:

1. A high level of external support from local health services and hospital management

2. Effective organisation, specifically strong clinical leadership and strategic and operational management.

Methods

Study design

Retrospective cross-sectional study of 19 clinical networks carried out using interviews, an internet survey and document review (Figure 1).

Figure 1.     Study overview with data collection and analysis components reported in this paper highlighted (click to enlarge)

Sample

Clinical networks in operation during a 3-year period (2006–2008) in NSW, Australia, covering: aged care, bone marrow transplantation, brain injury, cardiology, endocrinology, gastroenterology, gynaecological oncology, home enteral nutrition, neurosurgery, nuclear medicine, ophthalmology, radiology, renal medicine, respiratory medicine, severe burn injury, spinal cord injury, stroke, transition care and urology. The networks had a consistent organisational structure supported by the Agency executive, clinical co-chairs, a network manager at the operational level, and multidisciplinary network members.¹⁵ We assessed impacts on quality of care and system-wide change resulting from network activities during the study period to the end of 2011 to give sufficient time for changes to have occurred.

Managers and co-chairs of 19 networks were invited to participate in interviews to gather evidence for the primary outcomes. Contactable network members from 2006 to 2008 (n = 3234) – comprising medical, nursing and allied health professionals; consumers; nonhealth executives; and researchers and academics – were invited to participate in an internet survey.

Measures

A summary of outcome and explanatory variable definitions, their indicators and data collection methods is provided in Supplementary Files 1 and 2 (available from: hdl.handle.net/2123/17773).

The primary outcomes measured by this study were median ratings of impact on: 1) quality of care and 2) system-wide change. The secondary outcomes were development and implementation of quality improvement activities, clinician engagement, and perceived value of the network.

Explanatory factors measured were perceived external support; perceived leadership of the network manager, network co-chairs and the Agency executive; and strategic and operational management of the network.

Lastly, we collected data on descriptive and confounding variables: months of operation since the network establishment; network manager’s average full-time equivalent (FTE) working hours during the study period; average annual operating costs; and total in-kind contributions (i.e. sum of the cost of all people contributing to the network).

Data collection

Network managers and co-chairs during the study period were interviewed and asked to identify what, why and how impacts occurred as a result of network activities between 2006 and 2008. An impact had to: 1) meet the definition of quality of care and system-wide change; 2) be due to activities of the network; and 3) be corroborated by independent evidence. Supporting documentation was required to be submitted as evidence. A validation substudy was conducted to verify whether impacts identified in the interviews were attributable to network actions, and results demonstrated that the networks provided accurate information.²²

Expert panel rating of impacts (EXPAND method)

To obtain objective and standard measures of the two primary outcomes, an expert panel method was adapted from the RAND/UCLA (University of California, Los Angeles) appropriateness method; detailed description is reported elsewhere.^18,19 The EXPAND Method Panel consisted of five independent members with experience in quality improvement programs, implementing system-wide change, clinical care and the expert panel method. Panel members initially assessed the evidence of network impact to independently rate each network (premeeting ratings) on its impact on quality of care and system-wide change. Then a moderated face-to-face meeting was conducted, during which aggregated ratings were presented and discussed. At the conclusion of the meeting, each panel member independently re-rated each network (postmeeting ratings), the median of which was the final measure of network impact.

Internet survey

A survey was developed by building on existing clinical network measures, wider organisational literature, and findings of a qualitative prestudy.^10,18 The survey measured five explanatory factors: perceived external support; perceived leadership of the network manager, network co-chairs and the Agency executive; and strategic and operational management of the network. The survey also assessed secondary outcomes: perceived engagement of clinicians, and whether the network was perceived as valuable. The survey items had a five-point Likert response scale (‘strongly agree’ to ‘strongly disagree’, with an additional ‘don’t know’ option). A network-level measure was calculated as the mean of the individual scores for each domain. Cronbach’s alpha coefficients ranged from 0.75 to 0.92²⁰ across the seven domains, indicating acceptable to excellent construct validity.²³ Further details on the survey development, its psychometric properties and descriptive results have been published elsewhere.²⁰

Document review

Meeting minutes, records of quality improvement activities undertaken and financial records were audited using a standardised coding schedule and free-text annotations^15,18 to measure: one explanatory factor (strategic and operational management of a network); two secondary outcomes (developed and implemented quality improvement activities, engagement of clinicians); and the potential confounding factors.^15,18

Statistical methods

SAS 9.1 (Cary, NC: SAS Institute Inc) and Stata 11 software (College Station, TX: StataCorp) were used for analysis. The unit of analysis was the network. Relationships between outcomes, explanatory variables and confounders were examined using Spearman correlation coefficients for continuous explanatory variables and confounders, and t-tests for binary explanatory and confounder variables. Explanatory variables and confounding variables that had a correlation of 0.4 or more with the outcome were included in backward stepwise regression analyses and excluded if they had a p value of 0.1 or more. The regression analysis was first undertaken to investigate which explanatory variables were associated with the outcome. Potential confounders were then added to this model using the same backward stepwise selection process described above, but forcing the explanatory variables chosen in the previous model.

Data from a relevant Australian study²⁴ examining the association between clinical performance and organisational determinants in 19 healthcare organisations was used to estimate the likely effect size. Spearman correlation coefficients for associations of relevance to our study ranged from 0.45 to 0.71. With 19 networks and a 5% significance level, we had 80% power to detect a correlation coefficient as being statistically significant if it was 0.6 or more. Thus, we had sufficient power to detect moderate to large associations that are achievable and clinically meaningful.

Results

Interviews were conducted with 19 clinical network managers and 32 network co-chairs representing each clinical network. The internet survey was completed by 592 network members (18% response rate).

Descriptive results: characteristics of the clinical networks

Characteristics of the clinical networks are provided in Table 1 (published in more detail elsewhere).^15,18-20

Table 1.     Clinical network characteristics

Descriptive results: impacts of the networks on improving quality of care and system-wide change

Network impact on improving quality of care and facilitating system-wide change is summarised in Figure 2.

Nine networks (47%) had limited impact on improving quality of care, 37% (n = 7) had moderate impact and 16% (n = 3) had high impact. For facilitating system-wide change, 26% (n = 5) of networks had limited impact, 37% (n = 7) had moderate impact and 37% (n = 7) had high impact. An example of an improvement in quality of care was the development of care protocols to promote standardised approaches and eliminate variation. An example of system-wide change was the implementation of a new model of care for rural patients. Additional examples of impacts will be published elsewhere.¹⁹

Figure 2.     Networks’ impact on quality of care and system-wide change as assessed by expert panel (N = 19) (click to enlarge)

Predictors of the effectiveness of clinical networks to impact quality of care and system-wide change

Impact on quality of care

There were large to medium positive correlations between impact on quality of care and the explanatory factors: perceived leadership of network manager (r = 0.55; p = 0.016); perceived leadership of network co-chairs (r = 0.64; p = 0.003); perceived strategic and operational management of a network (r = 0.50; p = 0.029); and strategic and operational management of a network signified by number of meetings (r = 0.52; p = 0.022) (for further details, see Supplementary File 3, available from: hdl.handle.net/2123/17773). In regression analysis, perceived strategic and operational management of a network emerged as the only significant variable associated with impact on quality of care (coefficient estimate 0.86; 95% confidence interval [CI] 0.02, 1.69; p = 0.045) after controlling for network manager’s average FTE working hours (Table 2).

Table 2.     Predictors of the effectiveness of clinical networks to impact quality of care and system-wide change (summary of regression analyses^a)

CI = confidence interval; FTE = full-time equivalent; na = not applicable
^a Analyses of quality of care were adjusted for other variables in the model as indicated in the table, and also adjusted for confounders. Months of operation since establishment of the network and total in-kind contributions had low correlations with impact on quality of care (r < 0.4). Although average annual operating costs demonstrated a medium positive correlation with impact on quality of care (r = 0.54; p = 0.18), it was not retained in the final stepwise regression model.
^b Analyses of system-wide change were adjusted for other variables in the model as indicated in the table, and also adjusted for confounders. Months of operation since establishment of the network and total in-kind contributions had low correlations with impact on system-wide change (r < 0.4). Although average annual operating costs demonstrated a medium positive correlation with impact on system-wide change (r = 0.56; p = 0.13), it was not retained in the final stepwise regression model.

Impact on system-wide change

There were large to medium positive correlations between impact on system-wide change and the explanatory factors: perceived external support (r = 0.53; p = 0.019); perceived leadership of network manager (r = 0.50; p = 0.03); perceived leadership of network co-chairs (r = 0.59; p = 0.008); and strategic and operational management of a network signified by number of meetings (r = 0.71; p < 0.001) (for further details, see Supplementary File 3, available from: hdl.handle.net/2123/17773). In regression analyses, strategic and operational management of a network signified by number of meetings (coefficient estimate 0.23; 95% CI 0.06, 0.41; p = 0.013) and perceived leadership of the network manager (coefficient estimate 0.47; 95% CI 0.09, 0.85; p = 0.018) emerged as the significant factors associated with impact on system-wide change after controlling for network manager’s average FTE working hours (Table 2).

Secondary outcomes

Multivariable analyses showed that perceived strategic and operational management of a network was the only explanatory factor significantly associated with three of the secondary outcomes: developed and implemented quality improvement activities (coefficient estimate 5.87; 95% CI 1.03, 10.72; p = 0.021); engagement of clinicians – number of members (coefficient estimate 113.72; 95% CI 55.82, 171.61; p = 0.001); and network was perceived as valuable (coefficient estimate 0.47; 95% CI 0.06, 0.88; p = 0.026) after controlling for confounding factors (complete results for secondary outcomes available upon request).

Discussion

This study demonstrates that clinical networks can improve quality of care and facilitate system-wide change; however, we found substantial variation among NSW clinical networks. Only three of 19 networks demonstrated high impact on quality of care, and seven had high impact on system-wide change. Management and leadership of a clinical network, encompassing both strategic and operational elements, were the primary factors influencing the impact of clinical networks, as well as their ability to develop and implement quality improvement activities, engage clinicians, and be perceived as valuable. Corroborating evidence from a qualitative substudy examining network performance in detail indicated that charismatic and visionary leadership, as well as formal infrastructure to support network activities, were perceived as the most important factors for successful clinical networks.²¹

This research provided the scope to study clinical networks that covered divergent clinical areas across a large health system. It is the largest study of networks to date, as well as the first to quantitatively examine factors contributing to network success. Methodological innovations, including the systematic collection of data relating to objective and subjective measures, allowed sufficient standardisation of data across clinical areas for quantitative analyses. The EXPAND method enabled consistent assessment of network impacts, despite variation in clinical focus and the nature of desired impact, and could be adapted for use in other studies examining heterogeneous impacts arising in real-world research. The internet survey instrument developed is a valid tool applicable for use with clinical networks in other jurisdictions.²⁰ In addition, we conducted a qualitative substudy to further examine features of high- and low-impact networks to help interpret the quantitative results.²¹

Leadership, and strategic and operational management, were identified as key features that new and existing clinical networks should use to strengthen their operations and increase impact. Leadership, as measured in this study, encompassed aspects such as demonstrating vision and drive, and the ability to engage clinicians and build relationships with external stakeholders. Strategic and operational management measured in this study was defined by the organisational ability of the network manager; communication to assist with implementation; and a supportive, open environment with multidisciplinary representation. ‘Top down’ approaches to network management can sometimes stifle clinician engagement and innovation⁸; in contrast, a ‘bottom up’ approach can lack strategic planning, logistical efficiency and problem-solving capabilities that are possible with organisational-level planning and scaled implementation. However, a combined top down and bottom up approach to the design of clinical networks can provide strategic and operational support at the organisational level, while maintaining engagement at the clinical interface.

To optimise the effectiveness of clinical networks, it is recommended that strategic elements of management be established, such as systematic approaches to planning, and forming linkages to external parties to help implement quality improvement activities. In addition, operational elements such as the structure and organisation of network meetings, and communication and engagement strategies should be formalised. Future research should aim to prospectively measure change in the impact of clinical networks after implementing the recommendations from this study.

This study has some limitations. It was powered for analysis at the network level and multivariate subgroup analyses could not be conducted. Our impact measures were restricted to evidence available at the time of the study, and there may have been some measurement bias as a result. The survey response rate was less than the average for online surveys reported at 33%²⁵, and, although sensitivity analyses based on inverse probability weighting found correlation and regression results to be similar to the main (non-weighted) analyses, we cannot conclude that there was no response bias. It is possible that individuals with strong opinions were more likely to respond.²⁰ The external context was not significantly related to network impact. This may be due to lack of variation in external support between networks given their operation within one jurisdiction (NSW). Finally, after piloting, it became apparent that the available operational data were not of sufficient quality to measure all indicators identified in the initial protocol (for details, see Supplementary Files 1 and 2, available from: hdl.handle.net/2123/17773).

Conclusions

This study demonstrates that clinical networks can be vehicles to implement organisational change in healthcare, helping embed evidence based care into health systems and introduce quality improvement activities at a system-wide level. Clinical networks with strong management and leadership, combining top down and bottom up organisational approaches, are most successful. Clinicians and healthcare administrators across disciplines of medicine and healthcare internationally can use these findings to better organise formal and informal networks and increase their effectiveness in implementing evidence based healthcare for better patient outcomes.

Acknowledgements

The authors acknowledge the valuable contribution of the clinical network managers and co-chairs, and the Agency executive, for participating in this research. In addition, the authors thank Robert Sanson-Fisher, Daniel Barker and Mario D’Souza from the University of Newcastle for their valuable input into the study design, data management and data analysis stages, respectively. We acknowledge the support of Cyra Patel and Frances Gilham in assisting with preparing this manuscript.

As well as the named authors, the other members of the Clinical Networks Research Group are: Deanna Kalucy (Sax Institute), Rob Sanson-Fisher (University of Newcastle), Elizabeth McInnes (St Vincent’s Health and Australian Catholic University) and Cyra Patel (The Children’s Hospital at Westmead).

Dedication

As we were working on publishing this study, our colleagues Peter Castaldi and Kate Needham passed away. We would like to acknowledge the tremendous dedication and passion Peter and Kate had for clinical networks, clinical wellbeing and improving healthcare for all people.

Copyright:

© 2018 Haines et al. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.