In Practice

Testing a health research instrument to develop a state-wide survey on maternity care

Angela L Todd, Clare A Aitken, Jason Boyd, Maree Porter

Published 28 January 2016. doi: http://dx.doi.org/10.17061/phrp2611609
Citation: Todd AL, Aitken CA, Boyd J, Porter M. Testing a health research instrument to develop a state-wide survey on maternity care. Public Health Res Pract. 2016;26(1):e2611609.

  • Citation

  • PDF

About the author/s

Angela L Todd | Clinical and Population Perinatal Health Research, The Kolling Institute, The University of Sydney, NSW, Australia

Clare A Aitken | NSW Bureau of Health Information, Sydney, Australia

Jason Boyd | NSW Bureau of Health Information, Sydney, Australia

Maree Porter | Clinical and Population Perinatal Health Research, The Kolling Institute, The University of Sydney, NSW, Australia

Corresponding author

Angela L Todd | angela.todd@sydney.edu.au

Competing interests

None declared

Author contributions

AT and MP were responsible for the design and conduct of the research survey. CA and JB designed and undertook analysis of the survey tool. The manuscript was drafted by AT and CA, and reviewed by JB and MP. AT and JB drafted the revisions, which were reviewed by CA and MP.

Abstract

Partnerships between researchers and end users are an important strategy for research uptake in policy and practice. This paper describes how collaboration between an academic research organisation (the Kolling Institute) and a government performance reporting agency (the New South Wales [NSW] Bureau of Health Information) contributed to the development of a new state-wide maternity care survey for NSW.

Full text

Key points

  • Researchers at the Kolling Institute surveyed new mothers in New South Wales (NSW) public hospitals about their experiences of maternity care
  • Testing of the Kolling survey by the NSW Bureau of Health Information (BHI) demonstrated that it performed well on important survey performance measures. This analysis informed the BHI’s development of a state-wide maternity survey
  • Collaboration provided expertise that informed the development of the original survey and the new state-wide survey, which was implemented across NSW in 2015

Introduction

The primary policy governing maternity care in New South Wales (NSW) is ‘Towards normal birth in NSW’ (TNB).1 Central to the TNB policy is the promotion of ‘woman-centred care’ − care that recognises and responds to a woman’s social, emotional, physical, psychological, spiritual and cultural needs. Feedback from women about existing maternity services can help identify whether services are meeting women’s needs. Research conducted overseas and in some parts of Australia has explored women’s satisfaction with maternity care services, predominantly through dedicated surveys of women who have recently given birth.2-9 In NSW, the Bureau of Health Information (BHI) is the statutory agency responsible for reporting on the performance of the public health system, including monitoring patient experiences of hospital care. Maternity patients have previously been included in general overnight patient surveys in NSW, however these surveys have included only a small number of maternity-specific questions.10

In 2012, researchers at the Kolling Institute initiated discussions with the BHI and the NSW Ministry of Health, which has policy responsibilities for maternity services, about conducting a dedicated maternity survey. The researchers wanted to conduct policy-relevant research and were interested in better understanding women’s expectations of, and experiences with, maternity care. The TNB policy was scheduled for review in 2015, so feedback to policy makers from women about public maternity care services was considered important. The BHI’s expertise and experience in conducting patient surveys meant that it could contribute valuable advice and assistance to the project. Coincidentally, the BHI’s forward plan included the development of a new state-wide maternity survey, so the research survey (the ‘Kolling survey’) provided a timely opportunity to test a range of survey items and survey methods with a NSW maternity sample.

At the end of the survey data collection period the BHI approached the Kolling researchers, seeking information about the survey and lessons learnt. The BHI also asked for access to de-identified data to assess the performance of the survey tool. This article briefly describes the survey project and the analyses that were undertaken to assess specific measurement properties of the survey. We then describe how the findings were used to inform the design of the state-wide maternity survey, which was introduced in NSW in 2015.

Methods

Survey development

The Kolling survey included questions drawn from previous maternity surveys3,6,8 and consultations with stakeholders, including the BHI and the Ministry of Health. It comprised 123 questions structured around the three main maternity periods − antenatal, birth and postnatal. The survey also sought consent from each woman to link survey responses with health information recorded in each maternity unit’s clinical database (ObstetriX). Further details are available elsewhere.11 The study was approved by the NSW Population and Health Services Research Ethics Committee.

Survey sample

A sample of 2048 women, who represented all women giving birth at seven public maternity units in two local health districts in NSW between March and May 2013, was invited to participate in the survey. Letters were returned as undeliverable for 59 women (3%), leaving a sample of 1989, of whom 913 returned a completed survey.

Analysis of the survey tool’s measurement properties

The following measurement properties were examined:

  • Response rates and evidence of response bias
  • Quality of individual survey items, based on three measures
    • response dispersion, particularly ceiling or floor effects; that is, whether responses to an item were concentrated at one end of the response range – either overly positive (ceiling) or overly negative (floor)
    • data completeness – items with high levels of missing data (>4%) could indicate that questions and/or response options were not clear, not relevant, or difficult to answer
    • highly correlated items (r> 0.7), which could indicate duplication (i.e. measuring the same construct) or survey design issues (e.g. consecutive items not being treated as independent, resulting in coalescing of responses)
  • Data accuracy, which was assessed by comparing three data items (maternal age, mode of birth and neonatal outcome) that were replicated in the survey and available in ObstetriX. This comparison provided an indicator of external validity and how well women were responding to the survey questions.

All analyses were performed using SPSS version 22. Categorical survey items about the delivery of maternity care (‘performance-related items’) were recoded into a directional scale where optimal performance was represented as 100 and worst possible performance was represented as zero. Mean scores were calculated for the entire population response, and for that item in each maternity unit. Means were used to assess the ability of the survey items to effectively discriminate between the performances of the maternity units in the sample. Comparisons between means were undertaken using independent samples t-tests. Ceiling and floor effects were identified using three criteria: skewness >1.5, mean >85 and standard deviation r > 0.7 set as a ‘high correlation’. A p value of 0.05 was considered statistically significant.

Results

Response rates and response bias

The Kolling survey response rate was 46% (913/1989). Response rates varied by maternity unit, ranging from 34% to 59%. Nonresponders were younger than responders (mean 30.3 years, standard deviation [SD] 5.7 for nonresponders, compared with mean 31.9 years, SD 5.0 for responders; t (1988.01) = – 6.53, p < 0.001). Among women completing the survey, 97% (n = 882) gave consent to link survey data with clinical data recorded by the maternity units.

Quality of survey items

Response dispersion

Evidence of ceiling effects, but not floor effects, was found for some items (Table 1). Women gave high ratings to a set of questions about different aspects of their antenatal care (organisation of appointments, information and explanations provided, time to ask questions, treated with kindness and understanding, and involvement in decision making) that were grouped together in a grid; mean scores ranged from 87.4 to 95.6. Similar items were repeated in the birth and postnatal sections of the survey, and similar patterns of responses were found (birth mean scores 89.3−94.0; postnatal mean scores 84.4−89.7). Two other questions showed high levels of positive responses across all maternity units (‘skin-to-skin contact with baby’ and ‘contacted or visited at home by a midwife’).

Data completeness

Rates of missing responses for all items in the survey were ≤1%.

Highly correlated items

Two items about infant feeding (‘given active support and encouragement’ and ‘given practical help’) were highly correlated (r = 0.74, p < 0.001).

 

Table 1.     Maternity survey items with limited response dispersion

Maternity period Survey item Lowest hospital mean Highest hospital mean Mean score Standard error
Antenatal I was treated with kindness and understanding 93.2 100.0 95.6 0.5
Information was explained to me in a way I could understand 91.9 100.0 95.2 0.5
I was given enough time to ask questions or discuss my pregnancy 90.5 100.0 94.5 0.6
I felt comfortable asking questions 90.4 100.0 94.2 0.6
I felt involved in decisions about my care 89.9 100.0 92.6 0.7
I was given the information or explanations I needed 86.5 100.0 92.0 0.7
My antenatal appointments were well organised 84.1 100.0 87.4 0.8
My carer(s) gave me consistent information 83.3 100.0 87.9 0.8
Birth Did you have skin-to-skin contact with your baby shortly after the birth? 91.9 100.0 96.3 0.7
I was treated with kindness and understanding 89.4 100.0 94.0 0.6
Information was explained to me in a way I could understand 89.3 100.0 93.4 0.6
I felt comfortable asking questions 87.1 96.9 92.4 0.7
I felt involved in decisions about my care 84.5 100.0 89.3 0.8
I was given the information or explanations I needed 83.1 100.0 89.6 0.8
Postnatal Were you contacted or visited at home by a midwife? 95.3 100.0 98.0 0.4
I was treated with kindness and understanding 87.6 100.0 89.7 0.7
Information was explained to me in a way I could understand 87.3 100.0 89.5 0.8
I felt comfortable asking questions 83.8 100.0 86.7 0.8
I felt involved in decisions about my care 83.6 100.0 87.1 0.9
I was given the information or explanations I needed 82.4 96.9 84.4 0.9
I was given consistent information 74.6 100.0 77.8 1.1

Data accuracy

Comparison of three data items (maternal age, mode of birth and neonatal outcome) collected in the survey and in the ObstetriX database demonstrated high concordance. All but one survey respondent had a self-reported age up to 2 years different from the maternal age recorded in ObstetriX (a lag of 2 years was allowed between birth and the completion and return of the survey). For mode of birth, one woman reported having a caesarean section in the survey, which was recorded as an instrumental vaginal birth in ObstetriX. There were also 20 cases (2%) where survey respondents reported having a vaginal birth that was recorded as an assisted vaginal birth in ObstetriX. There was 100% concurrence between survey responses and ObstetriX data for ‘neonatal outcome’ (live birth, stillbirth or neonatal death).

Discussion

The analyses conducted by the BHI demonstrated that the Kolling survey tested well against several survey performance measures, with good response rates, minimal missing data, limited highly correlated items and accurate data capture. These results are all indicators of good measurement quality in a survey12,13 and have affirmed the use of the Kolling survey as a key resource in the BHI’s development of the state-wide NSW maternity survey.

The Kolling survey sought consent from women to link their survey data with existing routinely collected data. This has not been done in previous maternity surveys, although it has been used in other health-related research.14-16 The vast majority of women who returned a completed survey gave consent to record linkage (97%). Although we cannot assess whether the consent for linkage question dissuaded some women from responding to the Kolling survey, the level of consent is higher than typical for BHI patient surveys, which range from 79% (emergency department surveys) to 86% (paediatric surveys).

The development of the Kolling survey rested largely on two fundamental approaches: that it was evidence based, drawing on items used in previous maternity surveys; and that it was developed collaboratively with key stakeholders who provided opinions about survey items, advice on appropriate terminology and language, and methodological suggestions such as how best to present information to women about the request for consent for data linkage. The BHI was especially helpful in providing literature and advice about successful strategies for increasing survey response rates.

Collaboration with the BHI allowed testing of several measurement properties of the survey tool. This is an important process in survey development that is often done but less often publicly reported. The positive results for the Kolling survey have complemented the BHI’s usual rigorous survey design processes with the timely availability of a survey tool tested among a sample of NSW women.

This is a real and practical example of research being used to inform practice. The BHI and Kolling researchers have continued to collaborate on the new maternity survey, sharing some of the knowledge gained from the research survey that might inform the state-wide survey process, through joint membership of the NSW Maternity Care Survey Advisory Committee. The NSW Maternity Care Survey, implemented by the BHI in 2015, included 96 items, nearly half of which were either the same as or similar to items in the Kolling survey. Some items in the state-wide survey that were not in the Kolling survey were included because of findings from the Kolling study – for example, complementing an item about breastfeeding initiation with another about breastfeeding duration. The two highly correlated infant feeding items that were identified in the Kolling survey were not included in the state-wide survey but were replaced by others because the Kolling survey showed that infant feeding was an important issue for women. Some of the performance-related items in the Kolling survey that showed high ceiling effects were also discarded. In addition, the state-wide survey included various ‘standard’ items that the BHI uses in all of its patient experience surveys. Finally, based on the response patterns and correlations observed for questions presented in a grid structure in the Kolling survey, the NSW Maternity Care Survey presents all questions individually.

Regular interaction and effective partnerships between research producers and research users are promoted as important facilitators for research uptake in policy and practice17-19, and for building more extensive and ongoing collaborations.20 Our experience suggests that building such relationships is essential to the production of relevant and useful research. Our experience has also shown that serendipity and fortuitous connections between individuals can bring agencies together.

Conclusion

The development and testing of a maternity experience survey, initially developed for a research study, played a significant role in informing the development of the NSW Maternity Care Survey led by the BHI. The latter survey, introduced in 2015, gives 14 000 new mothers across NSW an opportunity to reflect on the maternity care they receive, and to identify areas of strength and opportunities for improvement. Health policy makers and service providers will, for the first time, have access to robust hospital-level information about maternity care in NSW and how well services are meeting women’s needs.

Acknowledgements

The original survey project was supported by a National Health and Medical Research Council Centre for Research Excellence Grant (1001066). The NSW Patient Survey Program is managed by the Bureau of Health Information on behalf of NSW Health.

Copyright:

Creative Commons License

© 2016 Todd et al. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.

References

  • 1. NSW Ministry of Health. Policy directive: maternity − towards normal birth in NSW. Sydney: NSW Government; 2010 [cited 2015 Nov 4]. Available from: www0.health.nsw.gov.au/policies/pd/2010/pdf/PD2010_045.pdf
  • 2. Public Health Agency of Canada. What mothers say: the Canadian maternity experiences survey. Ottawa: Public Health Agency of Canada; 2009 [cited 2015 Nov 4]. Available from: www.phac-aspc.gc.ca/rhs-ssg/pdf/survey-eng.pdf
  • 3. Redshaw M, Heikkila K. Delivered with care: a national survey of women’s experiences of maternity care. Oxford: National Perinatal Epidemiology Unit; 2010 [cited 2015 Nov 4]. Available from: www.npeu.ox.ac.uk/downloads/files/reports/Maternity-Survey-Report-2010.pdf
  • 4. Declercq E, Sakala C, Corry MP, Applebaum S, Herrlich A. Listening to mothers III: pregnancy and birth. New York: Childbirth Connection; 2013 [cited 2015 Nov 4]. Available from: transform.childbirthconnection.org/wp-content/uploads/2013/06/LTM-III_Pregnancy-and-Birth.pdf
  • 5. Care Quality Commission. National findings from the 2013 survey of women’s experiences of maternity care. Newcastle, UK: Care Quality Commission; 2013 [cited 2015 Nov 4]. Available from: www.cqc.org.uk/sites/default/files/documents/maternity_report_for_publication.pdf
  • 6. Miller YD, Thompson R, Porter J, Prosser SJ. Findings from the Having a Baby in Queensland survey 2010. Brisbane: Queensland Centre for Mothers & Babies, The University of Queensland; 2011 [cited 2015 Nov 4]. Available from: http://www.qcmb.org.au/media/pdf/Survey%20Report%202010.pdf
  • 7. Prosser SJ, Miller YD, Armanasco A, Hennegan J, Porter J, Thompson R. Findings from the Having a Baby in Queensland survey, 2012. Brisbane: Queensland Centre for Mothers & Babies, The University of Queensland; 2013 [cited 2015 Nov 4]. Available from: www.qcmb.org.au/media/pdf/HABIQ%20Survey%202012.pdf
  • 8. Brown S, Darcy M-A, Bruinsma F. Having a baby in Victoria 1989–2000: continuity and change in the decade following the Victorian Ministerial Review of Birthing Services. Aust N Z J Public Health. 2002;26(3):242−50. CrossRef | PubMed
  • 9. Brown S, Lumley J. Changing childbirth: lessons from an Australian survey of 1336 women. Br J Obstet Gynaecol. 1998;105(2):143−55. CrossRef | PubMed
  • 10. Ford JB, Hindmarsh D, Browne K, Todd AL. Are women birthing in NSW hospitals satisfied with their care? BMC Res Notes. 2015;8:106. CrossRef | PubMed
  • 11. Todd AL, Roberts CL, Porter M, Morris JM, Nicholl MC. Woman-centred maternity care: what do women say?  Protocol for a survey of women receiving maternity care in NSW. Sydney: University of Sydney; 2014 [cited 2015 Nov 4]. Available from: hdl.handle.net/2123/12108
  • 12. Alwin DF. Research on survey quality. Socio Meth Res. 1991;20(1):3−29. CrossRef
  • 13. Watson N, Wooden M. Assessing the quality of the HILDA survey wave 1 data. HILDA Project technical paper series No. 4/02. Canberra: Commonwealth Department of Family and Community Services; 2002 [cited 2015 Nov 4]. Available from: www.melbourneinstitute.com/downloads/hilda/Bibliography/HILDA_Technical_Papers/htec402.pdf
  • 14. Tate AR, Calderwood L, Dezateux C, Joshi H. Mother's consent to linkage of survey data with her child's birth records in a multi-ethnic national cohort study. Int J Epidemiol. 2006;35(2):294−8. CrossRef | PubMed
  • 15. Mindell J, Biddulph JP, Hirani V, Stamatakis E, Craig R, Nunn S, Shelton N. Cohort profile: the health survey for England. Int J Epidemiol. 2012;41(6):1585−93. CrossRef | PubMed
  • 16. Gray L, Batty GD, Craig P, Stewart C, Whyte B, Finlayson A, Leyland AH. Cohort profile: The Scottish Health Surveys Cohort: linkage of study participants to routinely collected records for mortality, hospital discharge, cancer and offspring birth characteristics in three nationwide studies. Int J Epidemiol. 2010;39(2):345−50. CrossRef | PubMed
  • 17. Lomas J, Fulop N, Gagnon D, Allen P. On being a good listener: setting priorities for applied health services research. Milbank Q. 2003;81(3):363−88. CrossRef | PubMed
  • 18. Haynes AS, Derrick GE, Chapman S, Redman S, Hall WD, Gillespie J, et al. From "our world" to the "real world": exploring the views and behaviour of the policy-influential Australian public health researchers. Soc Sci Med. 2011;72:1047−55. CrossRef | PubMed
  • 19. Moore G, Redman S, Haines M, Todd A. What works to increase the use of research in population health policy and programmes: a review. Evid Policy. 2011;7(3):277−305. CrossRef
  • 20. Ross S, Lavis J, Rodriguez C, Woodside J, Denis JL. Partnership experiences: involving decision-makers in the research process. J Health Serv Res Policy. 2003;8 Suppl 2:26−34. CrossRef | PubMed