How readable are Australian multilingual diabetes patient education materials? An evaluation of national English-language source texts March 2020, Volume 30, Issue 1

Introduction

In Australia, an estimated 1.2 million people had diabetes in 2014–15¹, and 2 million more people were at high risk of developing diabetes.² Diabetes is particularly prevalent in culturally and linguistically diverse (CALD) communities in Australia; the prevalence of diabetes in people born in eastern and southern Europe (14.5%), and North Africa and the Middle East (22.4%) is more than triple the national average.³ As a result, the Australian National Diabetes Strategy 2016–2020 recognises CALD groups as a priority population in strategic planning and implementation, and recommends tailored actions to meet the specific needs of CALD communities.⁴

The complex nature of diabetes makes self-management challenging. Its success is influenced by a number of factors, including health literacy. Education is an integral component of diabetes self-management.⁵ Given that education is often delivered using a written format, the reader’s health literacy skills (i.e. their ability to understand and act on the information) will be crucial to their self-management efforts. The only nationally representative survey to directly measure health literacy in Australia to date suggests that 60% of Australians have low health literacy, a disproportionate number of whom are from CALD backgrounds.⁶ Addressing health literacy has been identified as a strategic priority both nationally and internationally.⁷ In Australia, all diabetes health professionals are encouraged to address health literacy according to the joint position statement from the Australian Diabetes Educators Association and the Australian Government (via the National Diabetes Services Scheme [NDSS]).⁸

Health literacy can be addressed in several ways. A risk approach to health literacy prioritises removing literacy-related barriers in the environment and making health information easier to understand.⁹ Written patient education materials (PEMs) are a common means of communicating diabetes self-management information. From a risk perspective, ideal diabetes PEMs would employ health literacy principles and appropriately meet the needs of the target audience to support them to make more effective decisions and take appropriate health actions. One common approach is to write PEMs at an eighth-grade (or below) reading level.¹⁰ In addition, it has been recommended that sentences should be no more than 15 words in length.¹¹

Use of multilingual PEMs can improve the understanding of diabetes self-management practices among CALD groups. However, the utility of multilingual PEMs may be limited if the readability level is too high, because individuals may have low levels of literacy in their first language. Multilingual PEMs in Australia are normally prepared initially in English (‘source text’) and then translated into other languages “without distortion or omission and preserving the content and intent of the source message or text”.¹² Assessing the readability of source texts can provide a preliminary indication of the readability of translated language versions. To the authors’ knowledge, no research has looked into the readability of source texts that have been directly translated into other languages.

Aim

The purpose of this study was to explore the readability of the source texts for publicly available multilingual diabetes PEMs in Australia. Our primary aim was to evaluate whether these publicly available diabetes PEM source texts were written at the reading level recommended by health literacy guidelines.^10,11

Methods

Study design

This study used a non-experimental and descriptive design that involved calculating computerised readability scores of the source texts of selected PEMs. No ethical approval was required.

Inclusion and exclusion criteria for PEMs

Diabetes Australia is the key national diabetes authority in Australia. The NDSS is an Australian Government initiative that is administered with the assistance of Diabetes Australia to enhance the capacity of people with diabetes to understand and self-manage their life with diabetes. Diabetes Australia and the NDSS have jointly published PEMs on different topics. Both organisations are perceived as sources of authoritative information in this field, which is widely used in diabetes education.¹³

PEMs were included only if they were published by the NDSS and Diabetes Australia as a printable fact sheet (in PDF), publicly accessible from their official websites at no cost, and available in a language other than English. Information was excluded if it was only available in English; only available in video, pictorial or graphic form; or not directly related to diabetes self-management. The search was conducted in February–March 2017. The PEMs evaluated in this study were the versions published in 2016. Diabetes Australia and the NDSS updated the PEMs in October 2018, but the new materials were not included in this study.

Procedure

PEMs were categorised into four main content areas: general, nutritional, physical activity and medical information. The classification was completed independently by four senior bilingual diabetes educators. Disagreements were resolved through discussion.

All documents were then prepared in Microsoft Word by removing any text that was not in full sentences (including titles, headings, subheadings, short captions and bullet lists), embedded punctuation and document design elements (such as gaps, white spaces, pictures and images, and text boxes), as per published protocols for assessment by readability tools.¹⁴

The prepared text was assessed for readability using an automated online readability calculator. This is a cost-effective, time-effective and objective method to evaluate the literacy demand of written information materials.¹⁵ The calculator includes five commonly used readability indices: Flesch Kincaid Grade Level (FKGL), Gunning Fog Score (GFS), Coleman Liau Index (CLI), Simplified Measure of Gobbledygook Index (SMOG) and Automated Readability Index (ARI). These readability indices are used to assess health information, and have been recommended by health authorities and organisations.¹⁶ The readability calculator reports US grade reading levels, which can be used to estimate equivalent reading levels within the Australian educational system; according to the International Standard Classification of Education and the Australian Standard Classification of Education, US grade levels are estimated to be equivalent to Australian years of education in most states.¹⁷ The average score was calculated to provide a fairer prediction of the overall readability, because each measure predicts reading difficulty according to different linguistic variables; for example, the FKGL accounts for sentence length (number of words) and word length (number of syllables), whereas the SMOG calculates readability based on the number of polysyllabic words.¹⁸ See Table 1.

We also calculated the average number of words per sentence and the average number of syllables per word for each included PEM, as well as the proportion of complex terms (more than two syllables)¹⁹ and complex medical terms (medical terms with more than two syllables). A medical term was defined as a healthcare word or phrase that describes anatomy, diseases, procedures, medicines and experts in the field of health.²⁰ Final terms were checked against the medical terminology criteria in Taber’s Medical Dictionary Online.²¹ The terms matched in the search were considered medical terms. Prefaced medical expert terms (such as the provider registration status) were counted once, based only on the medical expert term (e.g. ‘registered podiatrist’ was counted as ‘podiatrist’). Medical terms that would be familiar to the target audience (e.g. Medicare, diabetes, obesity) and their medical abbreviations (including NDSS) were not counted. Complex medical terms were identified independently by three authors. Disagreements were resolved through discussion.

Data were entered and analysed using SPSS for Windows (Armonk, NY: IBM Corp, version 24.0).

 

Table 1.     Descriptions of readability measures

Measure	Linguistic variables	Formulaa
Flesch Kincaid Grade Level	Word length (syllables) Sentence length	0.39 × (words/sentences) + 11.8 × (syllables/words) – 15.59
Gunning Fog Score	Sentence length Complex words (3 syllables or more)	0.4 × ((words/sentences) + 100 × (complex words/words))
Simplified Measure of Gobbledygook Indexb	Sentence length Complex words	1.0430 ×√(30 x complex words/sentences) + 3.1291
Coleman Liau Index	Word length (characters) Sentence length	5.89 × (characters/words) – 0.3 × (sentences/words) – 15.8
Automated Readability Index	Word length (characters) Sentence length	4.71 × (characters/words) + 0.5 × (words/sentences) – 21.43

^a All calculations were conducted using the automated online readability calculator^15
b The Simplified Measure of Gobbledygook (SMOG) Index scores in this study (detailed in Table 2) were calculated using 10 sentences from each of the beginning, middle, and end of the text, as is standard procedure using the automated online readability calculator¹⁵

 

Results

Eight PEMs that met the inclusion criteria for this study were identified on the NDSS and Diabetes Australia websites. They covered a broad range of content relating to diabetes self-management (Table 2). All PEMs were available in 10 languages or more.

The average reading grade level of included PEMs was above Grade 10 (mean 10.4; standard deviation [SD] 0.9) (Table 2). On average, there were 1.5 syllables per word (mean 1.5; SD 0.1) and 17 words per sentence (mean 17.1; SD 0.9). The percentage of complex words ranged from 10.8% to 17.8% of the total number of words. The GFS and the CLI yielded the highest readability scores of all readability formulas, with readability scores above Grade 10 for each PEM. The average readability score was 10 for two-page PEMs, 11 for three-page PEMs, 10 for four-page PEMs and 11 for six-page PEMs (Table 3).

 

Table 2.     Description of patient education materials and readability statistics

Title	Content category	Availability in other languages	Pagesa	FKGLb	GFSb	SMOGb	CLIb	ARIb	Average readability scoreb	% of words that are complex	Average words per sentence	Average syllables per word
Understanding type 1 diabetes	General information	Arabic, simplified Chinese, traditional Chinese, Greek, Italian, Korean, Spanish, Turkish, Urdu, Vietnamese	2	8.7	12	8.8	11.5	8.4	10	14.2	16.0	1.5
Understanding type 2 diabetes	General information	Arabic, simplified Chinese, traditional Chinese, Greek, Italian, Korean, Spanish, Turkish, Urdu, Vietnamese	2	8.9	12	8.8	11.6	8.9	10	13.4	16.8	1.5
Understanding gestational diabetes	General information	Arabic, Bengali, simplified Chinese, traditional Chinese, Greek, Hindi, Italian, Korean, Samoan, Spanish, Turkish, Urdu, Vietnamese	3	10	13.2	9.7	13.3	10.5	11	16.1	17.2	1.6
Food choices for people with diabetes	Nutrition information	Arabic, simplified Chinese, traditional Chinese, Greek, Italian, Korean, Spanish, Turkish, Urdu, Vietnamese,	4	9.4	12.4	9.1	11.9	9.8	11	13.4	18.1	1.5
Hypoglycaemia	Medical information or instruction	Arabic, simplified Chinese, traditional Chinese, Greek, Italian, Korean, Spanish, Turkish, Urdu, Vietnamese	4	9.1	11.2	8.1	11.1	9	10	10.8	17.9	1.5
Diabetes complications	Medical information or instruction	Arabic, simplified Chinese, traditional Chinese, Greek, Italian, Korean, Spanish, Turkish, Urdu, Vietnamese	4	7.7	10.4	7.5	12	8.6	9	10.6	15.4	1.5
Pregnancy and diabetes	Medical information or instruction	Arabic, simplified Chinese, traditional Chinese, Greek, Italian, Korean, Spanish, Turkish, Urdu, Vietnamese	6	9.6	12.8	9.3	13.3	10.6	11	14.6	17.5	1.6
Physical activity	Physical activity information	Arabic, simplified Chinese, traditional Chinese, Greek, Italian, Korean, Spanish, Turkish, Urdu, Vietnamese	6	10.4	14	10.2	12.7	10.1	11	17.8	17.5	1.6
Averageb				9.2	12.3	8.9	12.2	9.5	10	13.8	17.1	1.5

ARI = Automated Readability Index; CLI = Coleman Liau Index; GFS = Gunning Fog Score; FKGL = Flesch Kincaid Grade Level; SMOG = Simplified Measure of Gobbledygook Index
^a One page is a standard A4-size sheet
^b The scores are equivalent to Australian grades

 

Table 3.    Average readability across patient education materials based on number of pages

Number of pages	Number of PEMS	FKGL	GFS	SMOG	CLI	ARI	Average
2	2	8.8	12.0	8.8	11.6	8.7	10
3	1	10	13.2	9.7	13.3	10.5	11
4	3	8.7	11.3	8.2	11.7	9.1	10
6	2	10	13.4	9.8	13	10.4	11

ARI = Automated Readability Index; CLI = Coleman Liau Index; GFS = Gunning Fog Score; FKGL = Flesch Kincaid Grade Level; PEM = patient education material; SMOG = Simplified Measure of Gobbledygook Index

 

In total, there were 95 unique complex medical terms across the eight PEMs, most commonly used for diseases, experts in the health field and medicines. Among those, there were 62 (65.3%) instances in which the term was not explained or defined after its first appearance (Table 4). The most frequently undefined medical terms were the medical expert terms (3–5 syllables), which occurred in all PEMs. The most frequently explained terms were ‘endocrinologist’ and ‘hypoglycaemia’, appearing three times across eight PEMs.

 

Table 4.     Number of complex medical terms in patient education materials and average readability

PEM title	Terms with definition	Terms without definition	Average readability	Number of unique complex medical terms			Proportion of complex medical terms without definition (%)
				Total	With definition	Without definition
Understanding type 1 diabetes	endocrinologist, insulin, ketoacidosis	antibodies, diabetes educator, dietitian, exercise physiologist, nurse practitioner, pancreas, podiatrist	10	10	3	7	70.0
Understanding type 2 diabetes	endocrinologist, haemoglobinA1c, insulin	antipsychotic, diabetes educator, dietitian, exercise physiologist, injectable medication, nurse practitioner, pancreas, pathology test, podiatrist, polycystic ovary syndrome, psychologist	10	14	3	11	78.6
Hypoglycaemia	Acarbose (Glucobay®) , glucagon, hypoglycaemia	diabetes educator, glucose jellybean, insulin, Lucozade	10	7	3	4	57.1
Diabetes complications	atherosclerosis, autonomic, cardiovascular, cataract, gastroparesis, genital, gingivitis, glaucoma, macrovascular, microvascular, neuropathy, ophthalmologist, peripheral arterial disease, peripheral neuropathy, retinopathy	amputation, cholesterol, diabetes educator, dietitian, optometrist, triglycerides	9	21	15	6	28.6
Pregnancy and diabetes	complication screening	contraception, dietitian, haemoglobin A1c, hypoglycaemia, insulin, intrauterine devices, miscarriage, nutrient supplement, pre-eclampsia, reversible methods	11	11	1	10	90.9
Physical activity	hypoglycaemia, peripheral vascular disease	cholesterol, diabetes educator, exercise physiologist, Medicare, muscular pain, osteoporosis, podiatrist	11	9	2	7	77.8
Understanding gestational diabetes	endocrinologist, gestational diabetes, hypoglycaemia	antipsychotic, Caesarean, diabetes educator, dietitian, exercise physiologist, insulin, metformin, nurse practitioner, obesity, pancreas, physiotherapist, placenta, polycystic ovary syndrome	11	16	4	12	75.0
Food choices for people with diabetes	HDL-cholesterol, LDL-cholesterol	cholesterol, dietitian, glycaemic index, insulin, triglycerides	115	7	2	5	71.4
Total	na	na	na	95	33	62	65.3
Average	na	na	10	11.9	4.1	7.8	na

na = not applicable; PEM = patient education material

 

Discussion

Summary of main findings

There is an ethical imperative to consider the readability of national multilingual diabetes PEMs, given that CALD groups have a higher prevalence of diabetes and poorer clinical outcomes than their Caucasian counterparts.²² The current study highlights that the national diabetes PEMs that serve as source texts for translation into multiple languages were written at a readability level higher than recommended, requiring above Year 10 Australian education to read. PEMs had an average sentence length greater than recommendations for readers with lower literacy.¹¹ They also contained a large number of complex medical terms, a high proportion (65.3%) of which were without in-text explanation or definition. Our findings suggest that the more pages a PEM contained, the higher its readability score (i.e. the lower its readability), although this was not tested for statistical significance.

Relationship to other studies

Although this was the first study to assess the readability of source texts (in English) for national multilingual diabetes PEMs in Australia, our findings are consistent with a large and growing body of literature that suggests that written information materials are written at a readability level that is too high for most patients.²⁰ An examination of culturally adapted English materials for African Americans, for example, showed that the mean readability of anticoagulation PEMs was Grade 11, when the reading skills of the target audience were between Grade 7 and Grade 8.²³ Together, these findings highlight the discrepancy between the skills and abilities of patients and the level of materials that are designed to support their care.

Poor readability of English-language source texts may also have implications for translated materials. Australian translators are guided by a code of ethics and conduct that emphasises preservation of the content and intent of a text.¹² Australian translators do not require accreditation to specialise in medical translation, and this may potentially lead to poor or inconsistent use of a medical corpus. Consequently, a good source text for any health education material lays the foundation for a good translated PEM, including from a health literacy perspective. Using a direct translation technique, it is possible that the translated PEMs (especially for the languages that have a similar lexical–semantic system to English) might have similarly poor readability.

This may be further exacerbated by additional translation issues. For example, other research examining the simplified Chinese version of the Diabetes Australia ‘Physical activity’ PEM found a number of inappropriate medical translation errors (e.g. ‘glucose gel’ mistranslated as ‘glucose jelly’).²⁴ Given that there is currently no universal medical corpus for translation, specific professional guidance for translating complex medical terms using health literacy principles is an important step for future research and practice.

Strengths and limitations

To date, our study is the first to evaluate the readability of national diabetes PEMs that have been translated for CALD populations. We employed five readability measures, and averaged scores across all measures as per recommended guidelines. We also examined factors (e.g. average number of words per sentence, average number of syllables per word, proportion of complex terms) that most strongly contribute to poor readability scores. Although only eight PEMs were analysed, the included PEMs were produced by the most reputable diabetes professional organisation in Australia, in conjunction with the Australian Government. As a result, they can be considered more influential than any other diabetes PEMs in this context.

This study only considered readability. A broad array of features beyond readability can make materials easier to understand, such as pictures, headers and visual cues. Although our findings may provide a proxy readability level for national multilingual diabetes PEMs in Australia, we did not assess the linguistic fluency (e.g. syntactic, phonological and lexical fluency) in the languages other than English and did not compare different language versions to examine differences in readability.

Future research directions

Additional work is needed to examine visual design, linguistic and structural features that promote or inhibit understanding in the source texts included in this study. Use of validated readability formulas for different languages would be useful to provide a deeper and richer view of the accessibility of multilingual PEMs. In addition, linguistic analyses would be useful to offer more sophisticated insights beyond the sentence level, and to provide more social–cultural understandings. It would also be valuable to evaluate the PEMS using cognitive interviews with consumers, to provider a deeper understanding of how meaning is extracted from the text.

In practice, producers of health materials must begin to consider the number of polysyllabic words, the use of complex medical terms and sentence length when developing PEMs, to reduce readability demands. Where complex medical terms are essential, they should be clearly defined (such as in a glossary of terms), or given simple explanations or descriptions when first introduced. It could also be worthwhile to systematically develop an Australian guide for medical translation (including a corpus) to assist decision making in translation of health materials. Lastly, value could be added by involving CALD consumers in the creation of content.

Conclusions

The English-language source texts for national multilingual diabetes PEMs examined in this study were of lower readability than recommended in health literacy guidelines. This was likely to be the result of use of polysyllabic words and complex medical terms, which are especially problematic when undefined. Improving readability of English-language source texts may help to ensure that the translated PEMs are more readable and accessible to their target readers. In conjunction with addressing other features that can make written materials easier to understand, this could better support diabetes self-management.

Acknowledgements

SL received a Research Training Program Stipend Scholarship from the Australian Government.The authors acknowledge the valuable contributions of Vania Khoury, Jennifer Zhen and Wenhui Zhang in categorising the diabetes PEMs, and the guidance Meng Ji has provided SL throughout the PhD research.

Peer review and provenance

Externally peer reviewed, not commissioned.

© 2020 Lin et al. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.