Disease or no disease? Disagreement on diagnoses between self-reports and medical records of adult patients

Abstract

Background: Previous studies reported moderate to good agreement between patients’ self-reported diseases and physicians’ registered diseases. Disagreement might hamper a good doctor–patient relationship and hamper good quality of care. Disagreement can be associated with demographic and psychosocial patient characteristics.

Objectives: To evaluate the level of agreement on reported chronic diseases between patients and their general practitioners (GPs); to assess whether disagreement relates to patient characteristics.

Methods: This study is embedded in a large GP based prospective cohort. Questionnaires of 2893 patients reporting on 14 chronic diseases are used. The agreement (percentage) between self-reported chronic diseases and the medical records was assessed first by descriptive statistics. To control for agreement by chance alone Cohen's kappa value was calculated. Type of (dis) agreement was further evaluated and associated with patient characteristics.

Results: Despite high agreement on diseases between patients and GPs, kappa's varied from 0.17 (inflammatory joint diseases and rheumatoid arthritis) to 0.86 (diabetes mellitus). Most often under-reporting and over-reporting was related to a decreased physical and mental quality of life and higher age.

Conclusion: kappa values between patients and GPs appeared to be low in this study.

Keywords::

• General practice
• medical records
• self-reported disease

KEY MESSAGE:

• kappa value between patients’ self-reported diseases and diseases as registered in GPs’ medical records was low in this study.

• This can hamper the doctor–patient relationship and good quality of care.

INTRODUCTION

In public health surveys or epidemiologic studies, patients are often asked about their medical history to determine an individual risk status or to assess the prevalence of diseases in a certain population (1). Although it is efficient to use these self-reports, the quality of data on self-reports is given little attention (2,3). Comparing the results of patients’ self-reports to medical records that are believed to be more valid (4,5) showed good agreement for diabetes, hypertension, pulmonary disease, cerebrovascular disease and myocardial infarction (2,6–10); while others found low agreement for heart failure, chronic bronchitis, chronic obstructive pulmonary disease (COPD), hypertension, osteoporosis and arthritis (1,2,6,9–12). In general, self-reports on medical conditions that are well defined and relatively easy to diagnose have a higher positive predictive value than conditions characterized by complex symptoms (7). The validity of self-reported chronic conditions by elderly responders is an important issue to study because of the methodological issues related to samples of older respondents with more chronic illnesses (12).

The accuracy of self-reporting diseases seems to depend on the participants’ knowledge and understanding of the relevant information, ability to recall it and the willingness to report disease (9,13). Merkin et al., found lower agreement between self-reported disease and medical records among males (1). Results regarding age are ambiguous (1,6,14). The presence of comorbid conditions was related to lower levels of agreement (1,6). To our knowledge only Kriegsman et al. evaluated the influence of depressive symptomatology, cognitive dysfunction and mobility limitations on the accuracy of self- reported disease (14).

Known barriers related to the physicians are: failure to diagnose the disease and/or not reporting the diagnosis to the patient (12).

In this study, agreement between self-reports of chronic diseases using questionnaire data and information from electronic medical records (EMR) from general practice is evaluated. Furthermore, the type of disagreement (i.e. over-reporting or under-reporting) will be related to demographic and psychosocial patient characteristics.

METHODS

Study design and population

A cross-sectional study based on data from the Study of medical information and lifestyles in Eindhoven (SMILE) in the Netherlands. SMILE is a large prospective cohort study in which self-reported aspects of health and lifestyle of people living in Eindhoven, the Netherlands are related to the electronic medical records (EMR) of participants’ general practitioners (GPs) (15). Questionnaires are posted to participants on an annual basis. The medical ethical committee of the Maastricht Academic Hospital has approved of the SMILE protocol (MEC 07–4–030) (15).

Medical records

Medical data was retrieved in November 2010 from Medicom^®, the electronic medical record system in which all participating GPs recorded all health related data on their patients. GPs in the Netherlands have a comprehensive view of the patients’ health status, because diagnoses made in hospitals are also incorporated in the GP's EMR. Disease episodes were coded using the International Classification of Primary Care (ICPC) (16,17). Lifetime prevalence of diseases was counted at the same time the questionnaire was completed (May 2010).

The questionnaire

In the SMILE questionnaire, a list of 14 chronic diseases was presented asking participants to record their actual health status with the following question, ‘Please indicate (yes/no) what disease or condition you have or have had?’ (see Appendix 1). This list was based on a frequently used medical screening questionnaire of the Dutch Association of General Practitioners (LHV) (18,19).

Demographic information from the questionnaire concerned information on sex, age, and education (low: primary school or lower vocational training; moderate: secondary school; high: high school, higher vocational training or university).

The presence of depression and/or anxiety was measured by the hospital anxiety and depression rating scale (HADS), which consists of two subscales each containing seven items on a four-point scale (from 0 to 3). Sum scores were calculated for each subscale representing a total score for anxiety and depression. A cut-off score of ≥ 8 for depression and anxiety was used (20).

Quality of life was measured by the SF-36. In the analyses, we used ‘total quality of life’, a subscale of the SF-36, composed of items regarding functional status, well-being, perceived health and general health. The sum score was transformed to a scale from 0 to 100 (21).

Statistical analysis

Before analysing the degree of agreement between self-reported (participant) and EMR data (GP) different sets of ICPC codes were defined in order to optimize the level of agreement. For example, self-reported ‘stroke or stroke-related effects’ were compared to ICPC K90 (cerebrovascular accident) as well as to the combination of K90 or K89 (transient cerebral ischaemia). For the final analysis, the ICPC codes as described in Appendix 1 were used. Without exception, these are the smallest spectra of ICPC codes.

The agreement (percentage) between self-reported chronic diseases and the medical records was assessed first by descriptive statistics. To control for agreement by chance alone Cohen's kappa value was calculated. Since the registration of chronic diseases in Dutch general practices generally appears to be valid, as compared to standard diagnostic criteria, with low numbers of false positive cases (14), we decided to use the EMR as the reference against which the self-reports are compared. We distinguished two types of disagreement; over-reporting: patients report a disease that is not recorded in their EMR; and under-reporting: patients did not report a disease, while it was recorded in their EMR (14).

Both under- and over-reporting were further explored by multiple logistic regression analyses (backward procedure) in two models, with either over- or under-reporting as a dependent variable, controlling for sex, age, number of chronic diseases in EMR and level of education (fixed block), and depression, anxiety, and quality of life as independent variables. All analyses were performed with SPSS (version 19.0).

RESULTS

Of the 3443 participants (response 83.4%) who responded to the questionnaire of May 2010, 2893 (84.0%) could be included in the analysis, excluding (n = 550) due to missing informed consent (n = 179) or missing data on the SF-36 and/or the HADS (n = 371). Analysis on baseline characteristics (sex, education, anxiety, depression, and quality of life) for those excluded did not show significant differences in comparison to the final study population. Mean age in the excluded group was slightly higher.

Baseline characteristics

The baseline characteristics of the participants are described in Table 1. The mean age of the participants was 68 ± 9 years; all levels of education were equally present. The mean quality of life score was 65 (SD 16), whereas 16% and 15% of the population were depressed and anxious, respectively.

Table 1. Baseline characteristics of participants (n = 2893, valid percentage).

	n	(%)
Male sex	1426	(49)
Age (years)
55–64	1107	(38)
65–74	1052	(37)
≥ 75	734	(25)
Education (n = 1202)
Low	379	(32)
Moderate	427	(35)
High	396	(33)
No. of chronic diseases
0	977	(34)
1	789	(27)
2	528	(18)
≥ 3	599	(21)
Anxiety (HADS)	439	(15)
Depression (HADS)	472	(16)
	Mean	(± SD)
Quality of Life (SF36)
Mental functioning	52	9
Physical functioning	46	10

Agreement and kappa values between patients and EMR

Agreement between participants’ reports and the EMRs showed an agreement of over 80% for most of the eleven chronic diseases (Table 2). Worst agreement was found for osteoarthritis (71%) and the best agreement for diabetes mellitus and stroke (both 96%). However, when controlling for agreement by chance alone, high kappa value (kappa ≥ 0.70) was only found for diabetes mellitus and malignancies. The analysis was not done for severe kidney disease, severe liver disease and epilepsy due to a limited number of cases.

Table 2. Over- and under-reporting of diseases, absolute numbers, and agreement between participants’ reports and the EMR^a (n = 2893, valid percentage).

	Patients %	EMR %	Agreement (%)	Kappa, (95% CI)			EMR Yes	No
Chronic lung disease	16	11	90	0.58 (0.62–0.54)	Patient	Yes	249	225	474
						No	61	2358	2419
							310	2538
Heart disease or MI	14	9	91	0.57 (0.62–0.52)	Patient	Yes	203	190	393
						No	64	2436	2500
							267	2626
Severe bowel disease	10	16	83	0.25 (0.30–0.21)	Patient	Yes	125	155	280
						No	326	2287	2613
							451	2442
Diabetes	14	15	96	0.86 (0.87–0.83)	Patient	Yes	372	34	406
						No	68	2419	2487
							440	2453
Malignancy/cancer	11	19	90	0.60 (0.64–0.55)	Patient	Yes	282	38	320
						No	264	2309	2573
							546	2347
Migraine	12	4	88	0.24 (0.30–0.19)	Patient	Yes	69	276	345
						No	58	2490	2548
							127	2766
Stroke or stroke-related	4	4	96	0.52 (0.59–0.44)	Patient	Yes	64	62	126
complaints						No	49	2718	2767
							113	2780
Inflammatory joint	19	24	72	0.17 (0.21–0.13)	Patient	Yes	222	336	558
diseases						No	483	1852	2335
							705	2188
Rheumatoid arthritis	8	3	91	0.17 (0.23–0.11)	Patient	Yes	34	204	238
						No	53	2602	2655
							87	2806
Osteoarthritis of knees, hips and hands	35	22	71	0.31 (0.34–0.27)	Patient	Yes	412	613	1025
						No	228	1640	1868
							640	2253
Severe back problems, hernia, osteoarthritis	26	9	77	0.24 (0.28–0.21)	Patient	Yes	172	574	746
						No	84	2063	2147
							256	2637

^aSevere kidney diseases, severe liver disease and epilepsy are missing because of low numbers.

Over- and under-reporting

Analysis of over- and under-reporting (Tables 3 and 4) showed that lower physical quality of life and less often lower mental quality of life, a higher number of chronic diseases and increasing age significantly contributed to both over- and under-reporting of many diseases. Other factors (i.e. sex, level of education, anxiety and depression) were associated with under- and over-reporting of specific diseases. For example, over-reporting diabetes: the discrepancy between self-reported diabetes that is not registered by the GP occurs 2.4 times more in anxious patients.

Table 3. Multivariate logistic regression analysis of over-reporting controlled for sex, age, level of education, and number of chronic diseases (from EMR), OR (95% CI).

Disease	Sex male vs female	Age 65–74 vs 55–64 years	Age 75 + vs 55–64 years	Education Intermediate vs low	Education High vs low	No. of diseases	QoL physical	QoL mental	Anxiety yes vs no
Chronic bronchitis, emphysema, asthma			0.66 (0.45–0.97)			0.89 (0.80–1.00)	0.97 (0.95–0.98)
Heart diseases or myocardial infarction	0.56 (0.41–0.76)	2.18 (1.43–3.33)	2.84 (1.85–4.38)				0.95 (0.94–0.97)	0.97 (0.96–0.99)	0.57 (0.34–0.94)
Severe bowel disease			1.63 (1.03–2.57)				0.97 (0.96–0.99)	0.96 (0.95–0.98)
Diabetes mellitus						1.27 (1.00–1.60)			2.43 (1.14–5.18)
Malignancy/cancer							0.97 (0.94–1.00)
Migraine	2.70 (2.05–3.56)			1.90 (1.12–3.25)	1.94 (1.11–3.38)		0.99 (0.97–1.00)	0.98 (0.97–1.00)
Stroke or stroke- related complaints							0.96 (0.94–0.98)	0.96 (0.94–0.98)
Inflammatory joint diseases						0.72 (0.64–0.81)	0.95 (0.94–0.97)	0.98 (0.97–0.99)
Rheumatoid arthritis	1.81 (1.31–2.46)				0.28 (0.12–0.64)	1.21 (1.09–1.35)	0.93 (0.92–0.94)	0.98 (0.97–0.99)
Osteoarthritis of knees, hips and hands	1.22 (1.08–1.47)					0.70 (0.64–0.76)	0.95 (0.94–0.96)	0.99 (0.98–1.00)
Severe back problems, hernia, osteoarthritis						0.91 (0.85–0.99)	0.95 (0.94–0.96)

Table 4. Multivariate logistic regression analysis of under-reporting controlled for sex, age, level of education, and number of chronic diseases (from EMR), OR (95% CI).

Disease	Sex male vs female	Age 65–74 vs 55–64 years	Age 75 + vs 55–64 years	Education Intermediate vs low	Education High vs low	No. of diseases	QoL physical	QoL mental	Depression yes vs no
Chronic bronchitis, emphysema, asthma	0.50 (0.29–0.85)	2.19 (1.05–4.59)				1.71 (1.45–2.01)
Heart diseases or myocardial infarction			4.50 (1.99–10.18)			1.68 (1.44–1.98)
Severe bowel disease				0.43 (0.27–0.70)	0.60 (0.38–0.94)	1.86 (1.70–2.04)	1.02 (1.00–1.03)
Diabetes mellitus			2.03 (1.03–4.0)			1.52 (1.30–1.77)
Malignancy/ cancer		1.79 (1.23–2.64)	2.98 (2.03–4.34)			1.76 (1.60–1.94)	1.02 (1.00–1.03)
Migraine	2.84 (1.56–5.18)				4.86 (1.29–18.25)	1.86 (1.55–2.23)
Stroke or stroke-related complaints	0.45 (0.24–0.83)					1.75 (1.46–2.09)			2.03 (1.09–3.79)
Inflammatory joint diseases	2.32 (1.79–3.00)					3.44 (3.07–3.85)	1.05 (1.04–1.07)	1.02 (1.01–1.03)
Rheumatoid arthritis						3.77 (2.94–4.84)
Osteoarthritis of knees, hips and hands	1.70 (1.23– 2.34)	0.61 (0.41–0.90)	0.62 (0.40–0.95)			2.84 (2.50–3.24)	1.06 (1.04–1.08)
Severe back problems, hernia, osteoarthritis		0.54 (0.30–0.96)				2.02 (1.74–2.33)

DISCUSSION

Main findings

This study of self-reported and registered medical data obtained in a large community-based cohort and taken from GP practices showed the highest kappa value for diabetes mellitus, and reasonable kappa values for malignancies, stroke, chronic lung disease and heart disease (including MI), respectively. For all other diseases, low kappa values were found. However, agreement was high for most diseases under study.

Interpretation of findings

Low kappa values might be the result of the low prevalence of the diseases, resulting in highly skewed distributions. Therefore, in this study kappa values should be interpreted with caution (22).

Diseases with more intensive treatment methods, like radiotherapy and chemotherapy for malignancies, and disease management programmes for diabetes mellitus or resulting in hospitalization are more likely to be reported by both patient and GP. In contrast, pain related conditions, like migraine and back problems are conditions that people often manage themselves without presenting it to their GP. Therefore, under-reporting in medical records or over-reporting by patients can easily occur. Our results are in line with previous studies that good agreement was related to well recognizable diseases that are relatively easily diagnosed (6,23,24).

Over-reporting is associated with a lower physical and mental quality of life. GPs might not always be aware of this relation, and could help patients by improving health information and by reassuring patients.

Under-reporting was related to a higher number of chronic diseases for all diseases. In contrast to over-reporting no associations were found with mental quality of life, indicating patients whom under-report are less worrisome than those over-reporting are. Possibly, complaints are addressed to old age. A better patient awareness of disease—e.g. in case of rheumatoid arthritis—could result in improved medication adherence and hence in better quality of life.

In this study, kappa was used to assess the correctness of patients’ reported diseases. We could have used sensitivity and specificity as indicators for the quality of self-reported diseases, but GPs’ information cannot be considered a true gold standard, possibly resulting in invalid outcomes (8,14). However, various studies have shown satisfactory validity and reliability of EMRs to support the results of this study (4,5).

Strengths and limitations

Overall, in this study kappa values appeared to be lower than expected when compared to previous studies (1,2,6,8). Our study population consisted of GPs’ patients, aged 55 years and older. This makes it hard to compare our results with other studies in which specific hospitalized patients are included or younger populations were used (1,6,8).

Registered data reflect routinely registration in daily practice. Medical records might lack some information, but at the same time, the available information is the basic input for medical decision making in daily general practice. Furthermore, the patient population enrolled in general practice is a good representation of the general population in the Netherlands, assuring the external validity of results. Moreover, patients excluded from the analyses because of non-response proved not to be a selective group.

Despite the relatively low kappa values, we tend to say that for health policy activities, patient reported diseases can be used on a population level, based on the high agreement since kappa values are hampered by the skewed distribution of diseases.

In this study, diseases were analysed in clusters, allowing a broad overview. This limits the identification of association on a diseases specific level.

Conclusion

In this study, in general we found a low kappa, but high agreement between medical records and patients’ reports, which can hamper the doctor–patient relationship and good quality of care.

Although electronic medical records are increasingly used for scientific research and quality indicators of health care, our results indicate that patient reported information can be used on a population level.

APPENDIX 1

In the SMILE questionnaire the following question was asked:

‘Please indicate what disease or condition you have or have had? (This question relates to your life.)’.

Appendix Table 1. Self-reported chronic diseases (questionnaire) and related ICPC-1 codes (EMR).

Liver disease or cirrhosis	D97
Severe bowel disease	D85, D86, D92, D93 and D94
Stroke or stroke-related complaints	K90
Heart diseases or myocardial infarction	K75, K76 and K77
Rheumatoid arthritis	L88
Inflammatory joint diseases	L88, L89, L90, L91, L92, L93 and L95
Osteoarthritis of knees, hips and hands	L88, L89, L90, L91, L95, L97
Severe back problems, hernia, sciatica or osteoarthritis	L83 and L86
Epilepsy	N88
Migraine	N89
Chronic bronchitis, emphysema and asthma	R91, R95 and R96
Diabetes mellitus	T90^a
Severe kidney disease	U70, U88
Malignancy or cancer (excluding skin malignancies)	A79, B72–74, D74–77, N74, N76, R84, R85, T71, T73, U75, U76, U77, W72, X75, X76, X77, Y77 and Y78

^aIn ICPC-1 T90 represents both type I and type II diabetes mellitus.

FUNDING

This study was performed as a thesis project and did not receive any additional funding.

ETHICS

The medical ethics committee of Maastricht University Hospital approved the SMILE study, which supplied the database used in this study (MEC 07–4–030).

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.