Advanced search

Reliability study of clinical electronic records with paper records in the NSW Public Oral Health Service March 2015, Volume 25, Issue 2

Angela V Masoe, Anthony S Blinkhorn, Kim Colyvas, Jane Taylor, Fiona A Blinkhorn

Published 30 March 2015. doi:
Citation: Masoe AV, Blinkhorn AS, Colyvas K, Taylor J, Blinkhorn FA. Reliability study of clinical electronic records with paper records in the NSW Public Oral Health Service. Public Health Res Pract. 2015;25(2):e2521519

  • Citation

  • PDF

About the author/s

Angela V Masoe | Faculty of Health and Medicine, School of Health Sciences, Oral Health, University of Newcastle, NSW, Australia

Anthony S Blinkhorn | Department of Population Oral Health, Faculty of Dentistry, University of Sydney, NSW, Australia

Kim Colyvas | School of Mathematical and Physical Sciences, University of Newcastle, NSW, Australia

Jane Taylor | Faculty of Health and Medicine, School of Health Sciences, Oral Health, University of Newcastle, NSW, Australia

Fiona A Blinkhorn | Faculty of Health and Medicine, School of Health Sciences, Oral Health, University of Newcastle, NSW, Australia

Corresponding author

Angela V Masoe | [email protected]

Competing interests

The findings from this study are those of the authors and do not reflect the views of the funding body or the NSW Ministry of Health. AM received travel support from the NSW Ministry of Health Rural and Remote Health Professionals Postgraduate Scholarship Scheme


Aim: Electronic health record (EHR) data have great potential for reuse in research and patient care quality improvement initiatives. However, in dual systems, where both electronic and paper health records are used, inconsistencies and errors may occur. The objective of this study was to determine the degree of agreement between EHR clinical data and paper records for reuse in clinical oral health research and quality improvement initiatives.

Methods: A random sample of 200 EHRs for adolescents from eight Area Health Services was obtained from the Information System for Oral Health New South Wales database of 29 599 records, and compared with 200 paper records for adolescents that were stored at clinics. The records were analysed for data reliability. The electronic records were percentage weighted to reflect the number of adolescents treated in each of the Area Health Services.

Results: The results showed an overall 95.0% agreement between the 200 individual EHRs and the 200 clinic-stored paper records. In 1.5% of cases, information contained in the paper record was not uploaded into the EHR, and in 3.5% of cases, information contained in the EHR was missing from the paper record.

Conclusions: It is possible to conclude that more deficiencies occurred in paper records compared with EHRs. These deficiencies should be taken into account if EHRs are to be reused for clinical oral health research or quality improvement initiatives. Considering the missing data and the great strides in information system technology, it would be logical to adopt one system, with a focus on electronic records to replace the paper records.

Full text

Key points

  • Electronic health record (EHR) systems in clinical facilities improve patient record control and documentation, and have great potential for reuse of data in oral health research and patient care quality improvement initiatives
  • Dual systems using EHR and paper records to record patients’ clinical oral health activities increase the risk of errors and can be inefficient
  • An electronic oral health record system to replace the current New South Wales Public Oral Health Service’s dual system is therefore recommended


Oral health information systems are essential for the evaluation and assessment of clinical dental services. If evaluation of these services is to be valid, then reliable, accurate patient records are required.1 The evolution of information systems and increased use of computers in clinical dentistry has placed more emphasis on the use of patient electronic health records (EHRs).2 A literature review by Hayrinen et al3 provided an EHR classification based on the International Organization for Standardisation classification, and highlighted the need for common terminology, especially when defining EHRs. Their paper defined an EHR as a repository of patient data in digital form, securely stored and transferable, made accessible upon request by different users with appropriate authorisation, and containing multilevel information for the efficient provision of integrated quality healthcare.

Benefits of EHRs in clinical facilities can include improved record control, efficient documentation, storage and access to patient data, and improved information for clinical management with quality data for appraisal of patient care.2 However, there are many different EHR systems, and this plethora of record types has caused confusion. This is especially true in the dental arena, which is a primary care system that collects huge amounts of data. Problems caused by different EHR systems led the World Health Organization (WHO) in the 1990s to call on its member states to ‘harmonise’ information systems for oral health to improve the quality of oral health structures and healthcare.1 The Chief Dental Officer for WHO and his team reported different information measurements and focuses across countries, including in recording and capturing various types of services; however, health outcomes were often excluded from reports. It was also noted that some countries had established outcome-oriented information systems, but these were not all compatible, leading to an inability to access planning data for national and international organisations.1 Well-designed EHRs can capture and enable comparison and analysis of patients’ and practitioners’ activities for quality improvement purposes.2

Atkinson2 found that a large percentage of electronic dental record data collected daily had the potential to be reused for research, generating new knowledge and improving patient care. Other researchers have demonstrated an increase in the reuse of electronic dental records for research, and have outlined common pitfalls associated with paper records, such as double handling of data and the likelihood of error in data transfer from paper to electronic databases.4,5 Additionally, it has been reported that parallel use of paper records and EHRs to capture medical and dental patient clinical data has resulted in inconsistencies between the record systems.4 On the other hand, it has been suggested that paper records are more flexible for describing patient symptoms and treatment, especially as some software packages are poorly designed and difficult to use.4 These issues are often raised to support the continuing use of paper records and as a justification for not implementing electronic data systems.

In the New South Wales (NSW) Public Oral Health Service, dual systems exist to administer and record clinical activity and manage patients’ dental care. The Information System for Oral Health (ISOH) is a centralised, state-wide repository for patient data in the NSW Public Oral Health Service, and is used to capture a clinician’s clinical activity as identified by dental treatment item numbers in the Australian national dental schedule.6 Paper records are predominantly used for recording a patient’s medical history, diagnosis, special test results, odontogram management plans and narratives of treatment provided. The current ISOH EHR performs some of these tasks (e.g. patient medical alerts are uploaded), but it does not have the capacity to record full medical histories. It is also possible to note ‘decayed’, ‘missing’ and ‘filled’ teeth scores, but the system lacks an odontogram for clinicians to record a detailed dental status assessment.

An overarching oral health record protocol from NSW Health provides guidance on the essential elements for the management of patients’ oral health records7, but there appears to be a lack of standard operational procedures for clinicians to seamlessly upload patient clinical activity into ISOH. This may affect data reliability for patient management and reuse in oral health research. There also appears to be a lack of information on the reliability of ISOH data and clinic-based paper records.

The aim of this reliability study was to investigate whether the EHR data from ISOH agreed with paper records maintained in clinics in terms of diagnostic and preventive care provided for adolescents. Dental therapists and oral health therapists are the main providers of oral healthcare for adolescents attending the NSW Public Oral Health Service. There is little information on whether therapists appropriately record the diagnostic and preventive clinical care they offer their patients in the EHR and paper records. This research topic was chosen because the same patient clinical activity (item) data has to be entered into the EHR and written (narrative/item) in the patient paper records for continuous care purposes. For the purposes of this study, the EHR is considered the benchmark standard, because it is used for clinician performance appraisals and dental treatment clinical indicators, and has potential to be reused in clinical oral health research and patient quality improvement initiatives.


The study assessed records dated between 1 January 2011 and 31 December 2011.

Electronic health records

The NSW State ISOH Manager drew a random sample of 200 EHRs from the total number (29 599) of adolescent diagnostic and preventive records. These were percentage weighted to reflect the number of adolescents treated by therapists in each of the eight Area Health Services (before the NSW Health restructure in 2011). Table 1 illustrates the sample distribution across the Area Health Services. The NSW Health restructure resulted in the formation of 15 (now 16) Local Health Districts (LHDs), which have maintained the same eight ISOH databases across districts, as per LHD service agreements.

Only diagnostic and preventive activities were included in the ISOH printout, as both are required to be entered into the EHR and the paper-based clinic record. Therapists were chosen as providers because of their workforce stability in the public health system over a one-year period compared with dentists.


Table 1.        Distribution of electronic health records analysed in the study, by Area Health Service

Area Health Service Number of records analysed % records analysed
North Coast 23 11.5
Northern Sydney and Central Coast 13 6.5
Hunter New England 37 18.5
Sydney South West 26 13.0
South Eastern Sydney and Illawarra Shoalhaven 36 18.0
South Western 27 13.5
Greater Western 15 7.5
Greater Southern 23 11.5
Total 200 100


Paper-based health records

For the paper-based records, the oral health clinical directors and service managers were contacted for the 16 LHDs where the selected EHRs had been entered. Approval was obtained to undertake clinic paper records comparison analysis across the 16 LHD dental clinics where records were stored with the EHR using an item number and activity audit tool. The audit tool was developed according to essential criteria outlined in the NSW Health Oral Health Record Protocol7, and was endorsed by two LHD clinical leaders. The audit tool was carefully pilot-tested by two dental assistants in two separate settings, both of whom were experienced in entering ISOH treatment activities. Adaptations to the audit tool were made based on their recommendations before it was used for this study.

Reliability assessment

LHD ISOH coordinators were chosen to undertake the reliability study, because they are not involved with clinical patient care, they have access to clinical records and the centralised LHD ISOH data, and they would collect the data at no cost. Face-to-face consultations and one video conference with LHD ISOH coordinators conducting the reliability exercise were undertaken to explain the study and offer advice about completing the assessment of both patient clinical record systems. Ten per cent of the data were further subjected to a second review by one oral health clinical leader and one oral health intake service coordinator, adhering to the reliability protocol process for consensus.

Assessment of whether the agreement between the EHRs and the paper-based records was satisfactory was made by considering the number/percentages of items/activities that disagreed between the two sets of 200 records. If both record systems did not have an entry for an item, this was counted as agreement between the record systems for that item.

The adequacy of the sample size was based on the uncertainty of percentage agreement. A Bayesian approach was used to determine the 95% credible intervals (credible intervals are the Bayesian equivalent of confidence intervals from frequentist statistics) over the range of 80% to 100%.8,9 At 80% agreement, the 95% credible interval was 73.9 to 84.9, giving a worst case of about a 6% uncertainty on the lower side. This was considered satisfactory for the purpose of the study. For a higher percentage agreement, the uncertainties were less – for example, at 95% agreement, the uncertainty was 91.0 to 97.2.8,9

The data were analysed using IBMSPSS Statistics10 and percentages and kappa values are used to describe key findings.

Ethics approval for the study was obtained from the Hunter New England Local Health District Lead Health and Research Ethics Committee (HREC) Reference No. 12/02/15/5.04, and the 16 LHDs. The Chief Health Officer, NSW Ministry of Health, approved the use of data for this investigation from the Centre for Oral Health Strategy, NSW ISOH.


The reliability study of ISOH data entries of EHR against paper record entries produced a 95.0% overall agreement rate.

In 1.5% of cases (= 33), information contained in the paper record was not included in the EHR; and in 3.5% of cases (= 78), information contained in the EHR was missing from the paper record (Table 2). The difference in agreement between the item numbers was significant (c2 (10) = 50.2, < 0.001). Using adjusted standardised residuals and follow-up chi-square tests, three different groups of agreement were identified. Items 012, 013, 121, 122 and 123 had similar agreement levels (combined agreement: 98.3%), followed by 011, 111 and 161 (combined agreement: 94.4%); 022, 131 and 141 had the lowest agreement (combined agreement: 89.8%).

Kappa agreement statistics were also calculated (Table 2). Nine kappa values were in the range 0.81–0.92, and the remaining two were 0.69 and 0.39. Combining all items gave a kappa value of 0.88.

When comparing item 011 ‘comprehensive oral examination’ in the EHR data with the paper record data, two cases had the item/activity entered on the paper record and not in the EHR, and seven cases had the item/activity entered in the EHR but not on the paper record. The percentage agreement of 95.5% was considered acceptable (Table 2).

Data were missing in paper records for radiographs (7.5% of cases) and fissure sealants (5.5% of cases). Inaccuracy in item number data entry by clinicians for the application of topical fluoride remineralising agents was also noted for both EHRs and paper records (Table 2).

The dietary advice error percentage for EHRs was 2.5% compared with 6.5% for the paper record, giving an overall agreement of 91.0% (kappa 0.82), which was below the acceptable threshold (Table 2), indicating an area requiring attention.

Oral hygiene instruction had an overall agreement of 88.5% (kappa 0.69), illustrating further inconsistencies (Table 2).


Table 2.        Reliability analysis of electronic health records and paper records

Item no. Item no. description Item no. (or activity) in both EHR and paper record (n) Item no. (or activity) not in EHR (n) Item no. (or activity) not in paper record (n) Agreement (%)a Kappa value (Κ)
011 Comprehensive oral examination 165 2 7 95.5 0.83
012 Periodic oral examination 9 2 2 98.0 0.81
013 Oral examination – limited 29 2 2 98.0 0.92
022 Radiographs (periapical and bitewing) 91 3 15 91.0 0.82
111 Professional clean 38 4 6 95.0 0.85
121 Topical application of fluoride remineralising agents 15 2 1 98.5 0.90
122 1 1 2 98.5 0.39
123 32 2 5 98.5 0.88
131 Dietary advice 87 5 13 91.0 0.82
141 Oral hygiene instruction 140 9 14 88.5 0.69
161 Fissure sealant 32 1 11 94.0 0.81
Total 639 33 78
Average 95.0 0.88
EHR = electronic health record
a  Agreement was based on N = 200 records, which included the records where both systems indicated an item was not provided. All kappa values were statistically significant.



The purpose of this study was to determine the degree of agreement between patient EHRs and paper records. The validation exercise was concerned only with whether the item number entered in the EHR correlated with what was written by the clinician in the paper record (narrative and/or item number) and vice versa. The findings of this reliability study illustrate inconsistencies, errors and missing data within the current dual record keeping system in the NSW Public Oral Health Service, suggesting that mistakes can occur when using dual record keeping systems. This has been reported by other researchers.5,11,12 For provision of continuous quality care for patients, it is disappointing to have levels of more than 10% inaccuracy in recording patient treatment care.

This study found missing data occurred more often in the paper records than the EHRs, demonstrating clinicians’ ease with uploading clinical activity into ISOH. Parallel use of dual systems is often used to meet an organisation’s various responsibilities.11 In the NSW Public Oral Health Service, the paper oral health record and EHRs contain legal patient clinical information and may be subpoenaed. However, the ISOH data entries are further linked to fiscal reports, key performance indicators for individual clinicians and LHD service agreements performance. It appears that clinicians have focused on entering data in the EHR, with less attention being placed on keeping accurate paper records. This may be because LHDs are required to record ‘weighted occasions of services’ generated from ISOH to meet state and Commonwealth activity-based targets, program measurements and individual clinical performance appraisals. These are critical and require all clinicians to enter all patient treatment data into the EHRs. This management focus on the EHRs may be due to in-built computer software triggers that assist and remind clinicians to upload essential patient data at certain points in the software, which the paper records lack.

Electronic patient and provider data have the potential to promote clinical practice quality improvement and research.12 Researchers have suggested that oral health professionals should develop a common record with standard codes, including clinical outcome measures, to make the EHR more useful for recording clinical treatments, facilitating research and improving quality of care.2 Nicholson et al’s13 paper, although in a pharmaceutical health setting and used in a different context, discussed the development and use of natural language processes that may enable the widespread use of free text in electronic records. There is scope for further research into the feasibility of inserting a free-text functionality into the current ISOH treatment module to support the clinical quantitative data.

This study shows that there is scope for a state-wide electronic oral health record to replace the paper system. This would reduce discrepancies and inconsistencies, and ensure that all patient clinical care activity is recorded accurately. The current ISOH database has functionalities that could be enhanced to capture medical histories and clinical treatment narratives, including expanding the current odontogram to include more detailed oral health charting. There is also an opportunity for research into how effective the ISOH system is in capturing clinicians’ community health preventive/promotion activities across NSW LHDs for better population oral health reporting. Additionally, it would be prudent to include health service researchers when developing a new system, so there can be an added value research component to monitor and assess patient care. There is great scope for harmonising electronic information systems to allow interstate comparison of public dental services in Australia.


This study found that diagnostic and preventive treatment data was inconsistent between the EHR and paper-based records, and missing to a greater extent in paper-based records than in EHRs. This should be accounted for if ISOH data are to be reused for clinical oral health research or patient quality improvement initiatives for prioritised populations. Considering the errors, missing data and great advances in technology, it would appear logical to undertake a pilot project to test a new electronic oral health record to replace the paper record in the NSW Public Oral Health Service.


Source of funding: the NSW Ministry of Health, Rural and Remote Allied Health Professionals Postgraduate Scholarship Scheme. Our appreciation to Mr Michael Jacobi, Manager Information Systems, NSW Centre for Oral Health Strategy, for data release. Special thanks to NSW LHD oral health clinical directors, health service managers, and ISOH coordinators for approval and for supporting the study.


Creative Commons License

© 2015 Masoe et al. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence, which allows others to redistribute, adapt and share this work non-commercially provided they attribute the work and any adapted version of it is distributed under the same Creative Commons licence terms.


  • 1. Petersen PE, Bourgeois D, Bratthall D, Ogawa H. Oral health information systems − towards measuring progress in oral health promotion and disease prevention. Bull World Health Organ. 2005;83(9):686−93. PubMed
  • 2. Atkinson JC. Electronic patient records for dental school clinics: more than paperless systems. J Dent Educ. 2002;66(5):634−42. PubMed
  • 3. Hayrinen K, Saranto K, Nykaren P. Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform. 2008;77(5):291−304.
  • 4. Schleyer T, Spallek H, Hernandez P. A qualitative investigation of the content of dental paper-based and computer-based patient record formats. J Am Med Inform Assoc. 2007;14(4):515−26. CrossRef | PubMed
  • 5. Mikkelsen G, Aasly J. Concordance of information in parallel electronic and paper based patient records. Int J Med Inform. 2001;63(3):123−31. PubMed
  • 6. Australian Dental Association. The Australian schedule of dental devices and glossary. 10th ed. Sydney: Australian Dental Association Inc; 2013 [cited 2015 Jan 28]: Available from:
  • 7. Centre for Oral Health Strategy. Oral health record protocols. Sydney: Department of Health, NSW; 2008 [cited 2013 Dec 12]. Available from:
  • 8. Gelman A, Carlin JB, Stern HS, Rubin DB. Bayesian data analysis. 2nd ed. Boca Raton, FL: Chapman & Hall/CRC; 2004.
  • 9. Carlin BP, Louis TA. Bayes and empirical Bayes methods for data analysis. London: Chapman & Hall; 1996.
  • 10. IBM SPSS Statistics vol 21 [Internet]. New York: IBM Corp; 2012.
  • 11. Stausberg J, Koch D, Ingenerf J, Betzler M. Comparing paper-based with electronic patient records: lessons learned during a study on diagnosis and procedure codes. J Am Med Inform Assoc. 2003;10(5):470−7. CrossRef | PubMed
  • 12. Song M, Liu K, Abromitis R, Schleyer T. Reusing electronic patient data for dental clinical research: a review of current status. J Dent. 2013;41(12):1148−63. CrossRef | PubMed
  • 13. Nicholson A, Tate AR, Koeling R, Cassell JA. What does validation of cases in electronic record databases mean? The potential for free text. Pharmacoepidemiol Drug Saf. 2011;20(3):321−4. PubMed