Testing a health research instrument to develop a state-wide survey on maternity care January 2016, Volume 26, Issue 1

Introduction

The primary policy governing maternity care in New South Wales (NSW) is ‘Towards normal birth in NSW’ (TNB).¹ Central to the TNB policy is the promotion of ‘woman-centred care’ − care that recognises and responds to a woman’s social, emotional, physical, psychological, spiritual and cultural needs. Feedback from women about existing maternity services can help identify whether services are meeting women’s needs. Research conducted overseas and in some parts of Australia has explored women’s satisfaction with maternity care services, predominantly through dedicated surveys of women who have recently given birth.^2-9 In NSW, the Bureau of Health Information (BHI) is the statutory agency responsible for reporting on the performance of the public health system, including monitoring patient experiences of hospital care. Maternity patients have previously been included in general overnight patient surveys in NSW, however these surveys have included only a small number of maternity-specific questions.¹⁰

In 2012, researchers at the Kolling Institute initiated discussions with the BHI and the NSW Ministry of Health, which has policy responsibilities for maternity services, about conducting a dedicated maternity survey. The researchers wanted to conduct policy-relevant research and were interested in better understanding women’s expectations of, and experiences with, maternity care. The TNB policy was scheduled for review in 2015, so feedback to policy makers from women about public maternity care services was considered important. The BHI’s expertise and experience in conducting patient surveys meant that it could contribute valuable advice and assistance to the project. Coincidentally, the BHI’s forward plan included the development of a new state-wide maternity survey, so the research survey (the ‘Kolling survey’) provided a timely opportunity to test a range of survey items and survey methods with a NSW maternity sample.

At the end of the survey data collection period the BHI approached the Kolling researchers, seeking information about the survey and lessons learnt. The BHI also asked for access to de-identified data to assess the performance of the survey tool. This article briefly describes the survey project and the analyses that were undertaken to assess specific measurement properties of the survey. We then describe how the findings were used to inform the design of the state-wide maternity survey, which was introduced in NSW in 2015.

Methods

Survey development

The Kolling survey included questions drawn from previous maternity surveys^3,6,8 and consultations with stakeholders, including the BHI and the Ministry of Health. It comprised 123 questions structured around the three main maternity periods − antenatal, birth and postnatal. The survey also sought consent from each woman to link survey responses with health information recorded in each maternity unit’s clinical database (ObstetriX). Further details are available elsewhere.¹¹ The study was approved by the NSW Population and Health Services Research Ethics Committee.

Survey sample

A sample of 2048 women, who represented all women giving birth at seven public maternity units in two local health districts in NSW between March and May 2013, was invited to participate in the survey. Letters were returned as undeliverable for 59 women (3%), leaving a sample of 1989, of whom 913 returned a completed survey.

Analysis of the survey tool’s measurement properties

The following measurement properties were examined:

• Response rates and evidence of response bias
• Quality of individual survey items, based on three measures
  • response dispersion, particularly ceiling or floor effects; that is, whether responses to an item were concentrated at one end of the response range – either overly positive (ceiling) or overly negative (floor)
  • data completeness – items with high levels of missing data (>4%) could indicate that questions and/or response options were not clear, not relevant, or difficult to answer
  • highly correlated items (r> 0.7), which could indicate duplication (i.e. measuring the same construct) or survey design issues (e.g. consecutive items not being treated as independent, resulting in coalescing of responses)
• Data accuracy, which was assessed by comparing three data items (maternal age, mode of birth and neonatal outcome) that were replicated in the survey and available in ObstetriX. This comparison provided an indicator of external validity and how well women were responding to the survey questions.

All analyses were performed using SPSS version 22. Categorical survey items about the delivery of maternity care (‘performance-related items’) were recoded into a directional scale where optimal performance was represented as 100 and worst possible performance was represented as zero. Mean scores were calculated for the entire population response, and for that item in each maternity unit. Means were used to assess the ability of the survey items to effectively discriminate between the performances of the maternity units in the sample. Comparisons between means were undertaken using independent samples t-tests. Ceiling and floor effects were identified using three criteria: skewness >1.5, mean >85 and standard deviation r > 0.7 set as a ‘high correlation’. A p value of 0.05 was considered statistically significant.

Results

Response rates and response bias

The Kolling survey response rate was 46% (913/1989). Response rates varied by maternity unit, ranging from 34% to 59%. Nonresponders were younger than responders (mean 30.3 years, standard deviation [SD] 5.7 for nonresponders, compared with mean 31.9 years, SD 5.0 for responders; t (1988.01) = – 6.53, p < 0.001). Among women completing the survey, 97% (n = 882) gave consent to link survey data with clinical data recorded by the maternity units.

Quality of survey items

Response dispersion

Evidence of ceiling effects, but not floor effects, was found for some items (Table 1). Women gave high ratings to a set of questions about different aspects of their antenatal care (organisation of appointments, information and explanations provided, time to ask questions, treated with kindness and understanding, and involvement in decision making) that were grouped together in a grid; mean scores ranged from 87.4 to 95.6. Similar items were repeated in the birth and postnatal sections of the survey, and similar patterns of responses were found (birth mean scores 89.3−94.0; postnatal mean scores 84.4−89.7). Two other questions showed high levels of positive responses across all maternity units (‘skin-to-skin contact with baby’ and ‘contacted or visited at home by a midwife’).

Data completeness

Rates of missing responses for all items in the survey were ≤1%.

Highly correlated items

Two items about infant feeding (‘given active support and encouragement’ and ‘given practical help’) were highly correlated (r = 0.74, p < 0.001).

Table 1.     Maternity survey items with limited response dispersion

Data accuracy

Comparison of three data items (maternal age, mode of birth and neonatal outcome) collected in the survey and in the ObstetriX database demonstrated high concordance. All but one survey respondent had a self-reported age up to 2 years different from the maternal age recorded in ObstetriX (a lag of 2 years was allowed between birth and the completion and return of the survey). For mode of birth, one woman reported having a caesarean section in the survey, which was recorded as an instrumental vaginal birth in ObstetriX. There were also 20 cases (2%) where survey respondents reported having a vaginal birth that was recorded as an assisted vaginal birth in ObstetriX. There was 100% concurrence between survey responses and ObstetriX data for ‘neonatal outcome’ (live birth, stillbirth or neonatal death).

Discussion

The analyses conducted by the BHI demonstrated that the Kolling survey tested well against several survey performance measures, with good response rates, minimal missing data, limited highly correlated items and accurate data capture. These results are all indicators of good measurement quality in a survey^12,13 and have affirmed the use of the Kolling survey as a key resource in the BHI’s development of the state-wide NSW maternity survey.

The Kolling survey sought consent from women to link their survey data with existing routinely collected data. This has not been done in previous maternity surveys, although it has been used in other health-related research.^14-16 The vast majority of women who returned a completed survey gave consent to record linkage (97%). Although we cannot assess whether the consent for linkage question dissuaded some women from responding to the Kolling survey, the level of consent is higher than typical for BHI patient surveys, which range from 79% (emergency department surveys) to 86% (paediatric surveys).

The development of the Kolling survey rested largely on two fundamental approaches: that it was evidence based, drawing on items used in previous maternity surveys; and that it was developed collaboratively with key stakeholders who provided opinions about survey items, advice on appropriate terminology and language, and methodological suggestions such as how best to present information to women about the request for consent for data linkage. The BHI was especially helpful in providing literature and advice about successful strategies for increasing survey response rates.

Collaboration with the BHI allowed testing of several measurement properties of the survey tool. This is an important process in survey development that is often done but less often publicly reported. The positive results for the Kolling survey have complemented the BHI’s usual rigorous survey design processes with the timely availability of a survey tool tested among a sample of NSW women.

This is a real and practical example of research being used to inform practice. The BHI and Kolling researchers have continued to collaborate on the new maternity survey, sharing some of the knowledge gained from the research survey that might inform the state-wide survey process, through joint membership of the NSW Maternity Care Survey Advisory Committee. The NSW Maternity Care Survey, implemented by the BHI in 2015, included 96 items, nearly half of which were either the same as or similar to items in the Kolling survey. Some items in the state-wide survey that were not in the Kolling survey were included because of findings from the Kolling study – for example, complementing an item about breastfeeding initiation with another about breastfeeding duration. The two highly correlated infant feeding items that were identified in the Kolling survey were not included in the state-wide survey but were replaced by others because the Kolling survey showed that infant feeding was an important issue for women. Some of the performance-related items in the Kolling survey that showed high ceiling effects were also discarded. In addition, the state-wide survey included various ‘standard’ items that the BHI uses in all of its patient experience surveys. Finally, based on the response patterns and correlations observed for questions presented in a grid structure in the Kolling survey, the NSW Maternity Care Survey presents all questions individually.

Regular interaction and effective partnerships between research producers and research users are promoted as important facilitators for research uptake in policy and practice^17-19, and for building more extensive and ongoing collaborations.²⁰ Our experience suggests that building such relationships is essential to the production of relevant and useful research. Our experience has also shown that serendipity and fortuitous connections between individuals can bring agencies together.

Conclusion

The development and testing of a maternity experience survey, initially developed for a research study, played a significant role in informing the development of the NSW Maternity Care Survey led by the BHI. The latter survey, introduced in 2015, gives 14 000 new mothers across NSW an opportunity to reflect on the maternity care they receive, and to identify areas of strength and opportunities for improvement. Health policy makers and service providers will, for the first time, have access to robust hospital-level information about maternity care in NSW and how well services are meeting women’s needs.

Acknowledgements

The original survey project was supported by a National Health and Medical Research Council Centre for Research Excellence Grant (1001066). The NSW Patient Survey Program is managed by the Bureau of Health Information on behalf of NSW Health.

Copyright:

© 2016 Todd et al. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.