Generating qualitative data by design: the Australian Longitudinal Study on Women’s Health qualitative data collection July 2016, Volume 26, Issue 3

Introduction

Qualitative research is both a vibrant and contested method of research.¹ It has become increasingly recognised as a way to explain complex social processes, and capture essential information about people’s beliefs, values and experiences that, in turn, influence particular health behaviours.² Often, free-text comments within the context of a survey are used to corroborate the content of the survey and to ensure that all relevant issues have been covered.³

Approaches to qualitative enquiry are highly diverse, and can include telephone, face-to-face or online exchanges; diary or journal keeping; photo records and observation; interviews conducted while walking, or at a person’s home or business; and analysis of images or personal narratives.

Narrative analysis, in particular, whereby people’s accounts or stories are interpreted and reported, has become popular as a qualitative research approach in response to the perceived lack of human stories in the 1960s.⁴ Data for narrative analysis can be collected using a variety of means, as long as it involves capturing an account, and can include invitations for people to record free-text comments on surveys. This approach can provide an opportunity for individuals to ‘tell their story’, and represents a rich source of data for understanding life experiences and specific phenomena, particularly in a longitudinal context.⁵

The Australian Longitudinal Study on Women’s Health (ALSWH) has been inviting women to write free-text comments on their surveys for the past 20 years. The ALSWH has a wide breadth and depth of qualitative information that demonstrates women’s life experiences. The purpose of this paper is to illustrate how qualitative free-text comments, collected within the context of a health survey, represent a rich data source for understanding specific phenomena.⁶

Methods

Australian Longitudinal Study on Women’s Health

The ALSWH first collected mailed survey data from more than 42 000 women in 1996 across three birth cohorts: 1973–78 (aged 18–23 in 1996), 1946–51 (aged 45–50) and 1921–26 (aged 70–75). A new cohort of women born between 1989 and 1995 was recruited in 2012–13. At the conclusion of each ALSWH survey, participants are asked “Have we missed anything?”, and are invited to record free-text comments on their health and any other topics that they felt the survey did not adequately include.

Selected analyses of free-text comments

Between 1996 and 2013, more than 56 000 free-text comments were contributed by more than 30 000 participants from the four ALSWH cohorts across a total of 22 surveys (Table 1).

Table 1. Number and percentage of participants who commented by cohort over each survey

ALSWH cohort (birth year)	Survey 1	Survey 2	Survey 3	Survey 4	Survey 5	Survey 6	Survey 7	Total number of words recorded by each cohort	% of cohort to have ever written a free-text comment in the ALSWH
1921–26	2978	4695	3955	4353	2481	698	–	1 221 713	59
1946–51	2392	2058	2672	2967	3731	3745	3021	1 155 298	61
1973–78	2423	1948	2266	2394	2415	1656	–	629 113	57
1989–95	1415	1150	740	–	–	–	–	106 924	16
Total	9208	9851	9633	9714	8627	6099	3021	3 113 048	na

– = survey not held; ALSWH = Australian Longitudinal Study on Women’s Health; na = not applicable

 

The sheer number of comments in the qualitative datasets can be daunting. Conversely, the fear that a respondent may not write about a topic of interest can be equally disturbing to researchers. However, a number of techniques have been used to manage the data.

Firstly, feasibility of topic areas can be facilitated by initial keyword searches – that is, searching the database for key terms to see whether any appropriate data are identified.

Secondly, it is useful to determine the broad inclusion and exclusion criteria of the data to be extracted – for example, identification of the cohorts to be included and the years when data were collected, and any experiences recorded in the quantitative data that need to be considered (e.g. participants with a particular condition might be included or excluded). Determining these criteria is the equivalent of determining a sampling frame from which a sample of data will be drawn.

Thirdly, the sample data can be extracted by selecting those data most relevant to the enquiry, or by determining keywords and only extracting those data that include the keywords. Although the latter option is more time- and cost-efficient, there is the potential to miss relevant data. For example, research conducted by Rich et al.⁷ using data from all three original cohorts showed that searching for the keyword ‘drought’ would have resulted in a loss of relevant data where terms such as ‘dry’ had been used to indicate drought. Nevertheless, the size of the data sample drawn and the resources available are likely to determine the methods used.⁸

To date, free-text comments on ALSWH surveys have been analysed to explore issues such as emotional distress, relationships, health-seeking behaviours and work–life balance. Numerous approaches have been taken to analyse ALSWH free-text comments, and selected examples are highlighted below.

Results

There were four main findings from our review of ALSWH analyses using women’s free-text comments, as detailed in the following results.

Combining inductive and deductive analysis

ALSWH data have been analysed both inductively (identifying emergent themes from the free-text comments) and deductively (looking for predetermined key issues or themes). The method chosen by Tavener et al.⁹, when exploring the phenomenon of life after stroke in older ALSWH participants, was a hybrid of a pre-existing framework of expected categories of stroke experience¹⁰ and a data-driven inductive approach.¹¹ The existing World Health Organization International Classification of Functioning, Disability and Health (ICF) core dataset for stroke was examined against older women’s free-text comments about living with stroke. The thematic analysis showed that, for older Australian women, concerns about post-stroke living were not adequately classified by the ICF, and that the women’s narratives of stroke experiences were not congruent with the clinical stroke phrasing, potentially impeding a full recovery for the women.

A hybrid approach was also taken by Tavener et al.¹² to explore older women’s longitudinal experiences of health and ageing. They applied the salutogenic model of wellness to explore adaptive strategies of women living with chronic conditions. Women’s free-text comments, recorded from 1999 to 2011 on 3-yearly ALSWH surveys, were subjected to longitudinal analysis for contrasting groups. Inductive thematic analysis identified categories of information described by the women, which were compared with deductive analysis findings of salutary expressions such as vulnerability, growth, coping and security. The hybrid approach enabled a rational and detailed longitudinal synthesis of women’s health experiences to be compared with the salutogenic model.

Pattern, theme and content analysis

Establishing categories or patterns of common textual meaning and/or counting the number of instances in which they are used are common ways to analyse free-text comments. Analyses can result in categorised representations of accounts, patterns of categories, or themes or particular narratives being identified, and a numerical description of textual features.

Thematic analysis of ALSWH data has explored the experiences of older women who take on caregiving roles.¹³ The analysis examined data from Survey 1 of the 1921–26 cohort, after 10% of respondents were identified from quantitative survey data as being caregivers and recorded free-text comments about their caregiving experiences. Following an empirical literature review on the topic¹⁴, which provided a conceptual framework for the content analysis to follow, researchers analysed the free-text comments of 168 women. This work identified that caregivers had significantly lower levels of emotional wellbeing than noncaregivers, and felt stressed, rushed and pressured. The women’s free-text comments provided insight into the forces that drive older women into providing family care.

A similar approach was repeated with the 1946–51 cohort.¹⁵ A content analysis of comments made by 185 women was examined for themes regarding caregiving. A cross-check step was included, where two researchers examined the comments, with one categorising the comments and the second recoding a subset. Emerging themes not already encompassed by the empirical literature search¹⁴ were then identified. Conducting a review of caregiving issues in older¹³ and then mid-aged women¹⁵ also allowed lifecourse nuances to emerge, such as differences in employment concerns, and caring for children and parents.

A thematic analysis of widowed women’s free-text comments identified that both short- and long-term effects followed the death of a husband.¹⁶ The work followed on from the findings of a quantitative examination of the health impacts of widowhood for the same cohort of women in the 12 months following bereavement¹⁷, which formed the theoretical framework for the qualitative analysis. Comments from more than 700 women elicited at Survey 1 of the 1921–26 cohort were examined. Key themes of health, social relationships and support, and financial and structural issues were identified as being of concern to the women.

Chojenta et al.¹⁸ used the free-text comments elicited by the 1973–78 cohort in Surveys 2–5 to identify risk factors for postnatal depression: examining those risk factors already identified, and exploring any new or emerging concepts in the qualitative data. The researchers analysed the initial content using the principles of directed content analysis, and codes were developed using deductive category application.¹⁹ Using the principles of directed content analysis, they used the results of previous research to identify initial categories. This approach enabled direct comparisons with the quantitative findings and consideration of the theoretical implications of the findings. The thematic analysis also involved a deeper level of analysis than the content. These findings indicate that, for some women, symptomatology might be a more suitable strategy for discussion by health practitioners rather than direct labelling of the event as ‘postnatal depression’.

Use of medicines was compared across three ALSWH cohorts by Walkom et al.²⁰, who conducted a thematic analysis of women’s experiences with day-to-day expenses that relate to medicines and their healthcare. Along with three broad themes of costs of medicines, doctor visits and complementary medicines, the researchers revealed age-specific issues with medicine costs, which included the importance of contraceptive medicines (for younger women), hormone replacement therapy (for mid-aged women) and osteoporosis medicines (for older women). This is critical information for life-stage affordability of health services, including access to medicines.

Longitudinal analyses of qualitative data

Building upon earlier work on ALSWH women’s experiences of widowhood, Kennaugh et al.²¹ conducted a longitudinal analysis of free-text comments to understand widowhood 15 years beyond the initial period of bereavement. The data were analysed inductively and deductively (as an extension of the work by Feldman et al.¹⁶). Two of the researchers reviewed the coding process, and the third author then reviewed Kennaugh’s coding against the full codebook. The differences were then reconciled by taking the strongest meaning held on each coded segment of text. Results confirmed that the original themes continued to persist over time, highlighting the importance of continuing to support women and to be responsive to their changing needs well after the initial period of bereavement.

Rich et al.⁷ used the free-text comments of women from the 1946–51 cohort to explore their experiences of living through drought. The use of longitudinal data in this context was ideal, because drought is known to be a long-lasting event, and the long-term impact of living through drought would be evident with longitudinal data. Women’s comments were examined in chronological order, guided by Riessman’s narrative analysis framework²² of discrete story and extended-account models.

Use of software to analyse data

Numerous computer-based software packages can assist with data organisation for qualitative data analysis. Software can help a researcher deal with large amounts of data, increase the flexibility of data organisation and allow auditability of the work.²³

Research by Tavener²⁴ made use of Leximancer software to contrast social narratives between groups of baby boomer women. Leximancer software uses word-association information to produce relationship and concept maps from excerpts of text.²⁵ For this work, the software was trialled as a ‘hands-off’ alternative approach to researcher-led inductive coding, to identify and describe the top three concepts identified from different groups of baby boomer women, categorised according to health, income and marital status. However, although the software did map women’s comments into different concepts of health, work and life, the primary researcher did not enjoy the hands-off experience and felt removed from the richness of the free-text comments. In contrast, Rich⁸ found the use of Leximancer to analyse free-text comments about the experience of drought across the original three ALSWH cohorts to be very useful, particularly for distinguishing the different themes that emerged over time for the three different cohorts.

ALSWH free-text comments have also been used to investigate the stability of propositional density (i.e. a measure of how much information is conveyed in an utterance or text) as a predictor of language decline with ageing.²⁶ More than 600 texts from the 1946–51 cohort were analysed longitudinally, using automated propositional density rating software. Results revealed propositional density to be a stable measure over time when comparing grouped data, and revealing between- and within-subject variation over time.

The above examples demonstrate the versatility of free-text data, which can be analysed using many of the techniques and tools that are used with more common qualitative data. Although the scope of the topics that have been investigated is wide, there remains a vast untapped mine of information in free-text comments in the ALSWH and many other survey-based studies.

Discussion

This paper has illustrated the value of free-text analysis and cohort longitudinal data. For example, differences exist in key themes between the ALSWH birth cohorts and over time.⁵ The authors concluded that free-text comments represent a valuable data source that is suitable for content, thematic and narrative analysis, particularly when examined longitudinally. Of particular significance is the fact that the women chose to record their comments on a research survey. The selection of stories chosen by the women to share with the ALSWH team reflects the relative level of importance of that information to them and their need to share that part of their experience. The women have written about a wide range of topics of interest to them, with both positive and adverse events recorded.

Although qualitative research methods are continually evolving, it is important that researchers continue to address issues of rigour – that is, reconsidering traditionally quantitative terms, such as validity and reliability, in qualitative terms instead. Rigour in qualitative research refers to the quality of the research process, evidenced by features such as credibility, dependability, confirmability and transparency.²⁷

For the ALSWH, the concept of rigour is considered in terms of the quality of the research process: from initial steps to anonymise the data through to iterations of coding and categorising, with the key being analytic reflexivity. Rigorous qualitative research must be both transparent and explicit. Researchers need to describe what they did in clear, simple language, account for their own influence on the work and be open about the analysis procedures. In this regard, analysis of ALSWH data is no different from any qualitative analysis.

Checklists encourage researchers to report results in a way that is considered both useful and credible. Following a systematic review of qualitative studies, Tong et al.²⁸ have published the consolidated criteria for reporting qualitative research (COREQ). The checklist aims to help researchers report key aspects of the research team, study methods and context, and analysis and interpretation, with reference to interviews and focus groups. Kitto et al.²⁹ published a checklist to address quality reporting across the domains of clarification, justification, procedural, rigour, representativeness, interpretation, reflexivity and transferability. Both the Tong and Kitto checklists have been used by researchers who analyse ALSWH free-text comments. However, we also acknowledge the caution offered by Barbour³⁰, to not reduce qualitative research to a “list of technical procedures” only. Furthermore, existing checklists tend to give consideration to the most common data collection methods, such as focus groups and interviews, and give less attention to methods that are equally valid but less common, such as diary studies and analysis of free-text comments (such as those collected by the ALSWH). This is a potential problem when seeking to demonstrate rigour to journal reviewers, if the methods chosen are not adequately dealt with in the recommended (or required) checklist.

Conclusion

It has become common practice to include an open-ended question at the conclusion of surveys, whether they are online, written or verbal. The purpose has largely been to gather feedback on the survey itself. ALSWH researchers have realised the potential of this resource, where participants can communicate with researchers without being confined to particular quantitative response options. These free-text comments are data in their own right, and have the potential to provide rich and valuable information about a wide variety of topics in the person’s own words.

Acknowledgements

The research on which this paper is based was conducted as part of the ALSWH at the University of Newcastle and the University of Queensland. We are grateful to the women who provided the survey data, the staff of the ALSWH, and the Australian Government Department of Health for funding. Researchers in the Research Centre for Generational Health and Ageing at the University of Newcastle are members of the Hunter Medical Research Institute.

Copyright:

© 2016 Tavener et al. This article is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.