Reliability study of clinical electronic records with paper records in the NSW Public Oral Health Service March 2015, Volume 25, Issue 2

Introduction

Oral health information systems are essential for the evaluation and assessment of clinical dental services. If evaluation of these services is to be valid, then reliable, accurate patient records are required.¹ The evolution of information systems and increased use of computers in clinical dentistry has placed more emphasis on the use of patient electronic health records (EHRs).² A literature review by Hayrinen et al³ provided an EHR classification based on the International Organization for Standardisation classification, and highlighted the need for common terminology, especially when defining EHRs. Their paper defined an EHR as a repository of patient data in digital form, securely stored and transferable, made accessible upon request by different users with appropriate authorisation, and containing multilevel information for the efficient provision of integrated quality healthcare.

Benefits of EHRs in clinical facilities can include improved record control, efficient documentation, storage and access to patient data, and improved information for clinical management with quality data for appraisal of patient care.² However, there are many different EHR systems, and this plethora of record types has caused confusion. This is especially true in the dental arena, which is a primary care system that collects huge amounts of data. Problems caused by different EHR systems led the World Health Organization (WHO) in the 1990s to call on its member states to ‘harmonise’ information systems for oral health to improve the quality of oral health structures and healthcare.¹ The Chief Dental Officer for WHO and his team reported different information measurements and focuses across countries, including in recording and capturing various types of services; however, health outcomes were often excluded from reports. It was also noted that some countries had established outcome-oriented information systems, but these were not all compatible, leading to an inability to access planning data for national and international organisations.¹ Well-designed EHRs can capture and enable comparison and analysis of patients’ and practitioners’ activities for quality improvement purposes.²

Atkinson² found that a large percentage of electronic dental record data collected daily had the potential to be reused for research, generating new knowledge and improving patient care. Other researchers have demonstrated an increase in the reuse of electronic dental records for research, and have outlined common pitfalls associated with paper records, such as double handling of data and the likelihood of error in data transfer from paper to electronic databases.^4,5 Additionally, it has been reported that parallel use of paper records and EHRs to capture medical and dental patient clinical data has resulted in inconsistencies between the record systems.⁴ On the other hand, it has been suggested that paper records are more flexible for describing patient symptoms and treatment, especially as some software packages are poorly designed and difficult to use.⁴ These issues are often raised to support the continuing use of paper records and as a justification for not implementing electronic data systems.

In the New South Wales (NSW) Public Oral Health Service, dual systems exist to administer and record clinical activity and manage patients’ dental care. The Information System for Oral Health (ISOH) is a centralised, state-wide repository for patient data in the NSW Public Oral Health Service, and is used to capture a clinician’s clinical activity as identified by dental treatment item numbers in the Australian national dental schedule.⁶ Paper records are predominantly used for recording a patient’s medical history, diagnosis, special test results, odontogram management plans and narratives of treatment provided. The current ISOH EHR performs some of these tasks (e.g. patient medical alerts are uploaded), but it does not have the capacity to record full medical histories. It is also possible to note ‘decayed’, ‘missing’ and ‘filled’ teeth scores, but the system lacks an odontogram for clinicians to record a detailed dental status assessment.

An overarching oral health record protocol from NSW Health provides guidance on the essential elements for the management of patients’ oral health records⁷, but there appears to be a lack of standard operational procedures for clinicians to seamlessly upload patient clinical activity into ISOH. This may affect data reliability for patient management and reuse in oral health research. There also appears to be a lack of information on the reliability of ISOH data and clinic-based paper records.

The aim of this reliability study was to investigate whether the EHR data from ISOH agreed with paper records maintained in clinics in terms of diagnostic and preventive care provided for adolescents. Dental therapists and oral health therapists are the main providers of oral healthcare for adolescents attending the NSW Public Oral Health Service. There is little information on whether therapists appropriately record the diagnostic and preventive clinical care they offer their patients in the EHR and paper records. This research topic was chosen because the same patient clinical activity (item) data has to be entered into the EHR and written (narrative/item) in the patient paper records for continuous care purposes. For the purposes of this study, the EHR is considered the benchmark standard, because it is used for clinician performance appraisals and dental treatment clinical indicators, and has potential to be reused in clinical oral health research and patient quality improvement initiatives.

Method

The study assessed records dated between 1 January 2011 and 31 December 2011.

Electronic health records

The NSW State ISOH Manager drew a random sample of 200 EHRs from the total number (29 599) of adolescent diagnostic and preventive records. These were percentage weighted to reflect the number of adolescents treated by therapists in each of the eight Area Health Services (before the NSW Health restructure in 2011). Table 1 illustrates the sample distribution across the Area Health Services. The NSW Health restructure resulted in the formation of 15 (now 16) Local Health Districts (LHDs), which have maintained the same eight ISOH databases across districts, as per LHD service agreements.

Only diagnostic and preventive activities were included in the ISOH printout, as both are required to be entered into the EHR and the paper-based clinic record. Therapists were chosen as providers because of their workforce stability in the public health system over a one-year period compared with dentists.

Table 1.        Distribution of electronic health records analysed in the study, by Area Health Service

Paper-based health records

For the paper-based records, the oral health clinical directors and service managers were contacted for the 16 LHDs where the selected EHRs had been entered. Approval was obtained to undertake clinic paper records comparison analysis across the 16 LHD dental clinics where records were stored with the EHR using an item number and activity audit tool. The audit tool was developed according to essential criteria outlined in the NSW Health Oral Health Record Protocol⁷, and was endorsed by two LHD clinical leaders. The audit tool was carefully pilot-tested by two dental assistants in two separate settings, both of whom were experienced in entering ISOH treatment activities. Adaptations to the audit tool were made based on their recommendations before it was used for this study.

Reliability assessment

LHD ISOH coordinators were chosen to undertake the reliability study, because they are not involved with clinical patient care, they have access to clinical records and the centralised LHD ISOH data, and they would collect the data at no cost. Face-to-face consultations and one video conference with LHD ISOH coordinators conducting the reliability exercise were undertaken to explain the study and offer advice about completing the assessment of both patient clinical record systems. Ten per cent of the data were further subjected to a second review by one oral health clinical leader and one oral health intake service coordinator, adhering to the reliability protocol process for consensus.

Assessment of whether the agreement between the EHRs and the paper-based records was satisfactory was made by considering the number/percentages of items/activities that disagreed between the two sets of 200 records. If both record systems did not have an entry for an item, this was counted as agreement between the record systems for that item.

The adequacy of the sample size was based on the uncertainty of percentage agreement. A Bayesian approach was used to determine the 95% credible intervals (credible intervals are the Bayesian equivalent of confidence intervals from frequentist statistics) over the range of 80% to 100%.^8,9 At 80% agreement, the 95% credible interval was 73.9 to 84.9, giving a worst case of about a 6% uncertainty on the lower side. This was considered satisfactory for the purpose of the study. For a higher percentage agreement, the uncertainties were less – for example, at 95% agreement, the uncertainty was 91.0 to 97.2.^8,9

The data were analysed using IBMSPSS Statistics¹⁰ and percentages and kappa values are used to describe key findings.

Ethics approval for the study was obtained from the Hunter New England Local Health District Lead Health and Research Ethics Committee (HREC) Reference No. 12/02/15/5.04, and the 16 LHDs. The Chief Health Officer, NSW Ministry of Health, approved the use of data for this investigation from the Centre for Oral Health Strategy, NSW ISOH.

Results

The reliability study of ISOH data entries of EHR against paper record entries produced a 95.0% overall agreement rate.

In 1.5% of cases (n = 33), information contained in the paper record was not included in the EHR; and in 3.5% of cases (n = 78), information contained in the EHR was missing from the paper record (Table 2). The difference in agreement between the item numbers was significant (c² (10) = 50.2, p < 0.001). Using adjusted standardised residuals and follow-up chi-square tests, three different groups of agreement were identified. Items 012, 013, 121, 122 and 123 had similar agreement levels (combined agreement: 98.3%), followed by 011, 111 and 161 (combined agreement: 94.4%); 022, 131 and 141 had the lowest agreement (combined agreement: 89.8%).

Kappa agreement statistics were also calculated (Table 2). Nine kappa values were in the range 0.81–0.92, and the remaining two were 0.69 and 0.39. Combining all items gave a kappa value of 0.88.

When comparing item 011 ‘comprehensive oral examination’ in the EHR data with the paper record data, two cases had the item/activity entered on the paper record and not in the EHR, and seven cases had the item/activity entered in the EHR but not on the paper record. The percentage agreement of 95.5% was considered acceptable (Table 2).

Data were missing in paper records for radiographs (7.5% of cases) and fissure sealants (5.5% of cases). Inaccuracy in item number data entry by clinicians for the application of topical fluoride remineralising agents was also noted for both EHRs and paper records (Table 2).

The dietary advice error percentage for EHRs was 2.5% compared with 6.5% for the paper record, giving an overall agreement of 91.0% (kappa 0.82), which was below the acceptable threshold (Table 2), indicating an area requiring attention.

Oral hygiene instruction had an overall agreement of 88.5% (kappa 0.69), illustrating further inconsistencies (Table 2).

Table 2.        Reliability analysis of electronic health records and paper records

EHR = electronic health record
a  Agreement was based on N = 200 records, which included the records where both systems indicated an item was not provided. All kappa values were statistically significant.

Discussion

The purpose of this study was to determine the degree of agreement between patient EHRs and paper records. The validation exercise was concerned only with whether the item number entered in the EHR correlated with what was written by the clinician in the paper record (narrative and/or item number) and vice versa. The findings of this reliability study illustrate inconsistencies, errors and missing data within the current dual record keeping system in the NSW Public Oral Health Service, suggesting that mistakes can occur when using dual record keeping systems. This has been reported by other researchers.^5,11,12 For provision of continuous quality care for patients, it is disappointing to have levels of more than 10% inaccuracy in recording patient treatment care.

This study found missing data occurred more often in the paper records than the EHRs, demonstrating clinicians’ ease with uploading clinical activity into ISOH. Parallel use of dual systems is often used to meet an organisation’s various responsibilities.¹¹ In the NSW Public Oral Health Service, the paper oral health record and EHRs contain legal patient clinical information and may be subpoenaed. However, the ISOH data entries are further linked to fiscal reports, key performance indicators for individual clinicians and LHD service agreements performance. It appears that clinicians have focused on entering data in the EHR, with less attention being placed on keeping accurate paper records. This may be because LHDs are required to record ‘weighted occasions of services’ generated from ISOH to meet state and Commonwealth activity-based targets, program measurements and individual clinical performance appraisals. These are critical and require all clinicians to enter all patient treatment data into the EHRs. This management focus on the EHRs may be due to in-built computer software triggers that assist and remind clinicians to upload essential patient data at certain points in the software, which the paper records lack.

Electronic patient and provider data have the potential to promote clinical practice quality improvement and research.¹² Researchers have suggested that oral health professionals should develop a common record with standard codes, including clinical outcome measures, to make the EHR more useful for recording clinical treatments, facilitating research and improving quality of care.² Nicholson et al’s¹³ paper, although in a pharmaceutical health setting and used in a different context, discussed the development and use of natural language processes that may enable the widespread use of free text in electronic records. There is scope for further research into the feasibility of inserting a free-text functionality into the current ISOH treatment module to support the clinical quantitative data.

This study shows that there is scope for a state-wide electronic oral health record to replace the paper system. This would reduce discrepancies and inconsistencies, and ensure that all patient clinical care activity is recorded accurately. The current ISOH database has functionalities that could be enhanced to capture medical histories and clinical treatment narratives, including expanding the current odontogram to include more detailed oral health charting. There is also an opportunity for research into how effective the ISOH system is in capturing clinicians’ community health preventive/promotion activities across NSW LHDs for better population oral health reporting. Additionally, it would be prudent to include health service researchers when developing a new system, so there can be an added value research component to monitor and assess patient care. There is great scope for harmonising electronic information systems to allow interstate comparison of public dental services in Australia.

Conclusion

This study found that diagnostic and preventive treatment data was inconsistent between the EHR and paper-based records, and missing to a greater extent in paper-based records than in EHRs. This should be accounted for if ISOH data are to be reused for clinical oral health research or patient quality improvement initiatives for prioritised populations. Considering the errors, missing data and great advances in technology, it would appear logical to undertake a pilot project to test a new electronic oral health record to replace the paper record in the NSW Public Oral Health Service.

Acknowledgements

Source of funding: the NSW Ministry of Health, Rural and Remote Allied Health Professionals Postgraduate Scholarship Scheme. Our appreciation to Mr Michael Jacobi, Manager Information Systems, NSW Centre for Oral Health Strategy, for data release. Special thanks to NSW LHD oral health clinical directors, health service managers, and ISOH coordinators for approval and for supporting the study.

Copyright:

© 2015 Masoe et al. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.