A Validation of the Swedish Colorectal Cancer Register – With Focus on Histopathology, Complications and Recurrences

Abstract

Background

There is an urgent need to evaluate the quality of healthcare systems to improve and deliver high-quality care. Clinical registries have become important platforms for performance measurements, improvements, and clinical research. Hence, the quality of data in registries is crucial. This study aimed to assess the validity of data in the Swedish Colorectal Cancer Register (SCRCR).

Methods

Seven hundred patients from 12 hospitals were randomly selected and proportionally distributed among three different hospital categories in Sweden using two-stage cluster sampling. Validity was assessed by re-abstracting data from the medical files of patients reported to the SCRCR in 2015. Data on histopathology, postoperative complications, and a 3-year follow-up were selected for validation. Re-abstracted data were defined as source data, and validity was defined as the proportion of cases in the SRCRC dataset that agreed with the source data. Validity was expressed as the percentage of exact agreement of non-missing data in both data sets, and Cohen´s kappa coefficient (κ) was used to measure the strength of the agreement.

Results

The median agreement of the categorical histopathology variables was 93.4% (κ = 0.83). The general postoperative complication variable showed substantial agreement (84.3%, κ = 0.61). Likewise, the variable for overall cancer recurrence showed an almost perfect agreement (95.7%, κ = 0.86), whereas specific variables for local recurrence and distant recurrence displayed only moderate and fair agreement (85.9% and 89.1%, κ = 0.58 and 0.34, respectively).

Conclusion

Validation of the SCRCR data showed high validity of pathology data and recurrence rates, whereas detailed data on recurrence were not as good. Data on postoperative complications were less reliable, although the incidence and Clavien–Dindo grading of severe complications (grade 3b or higher) were reliable.

Keywords:

• quality registry
• national registry
• validation
• colorectal cancer
• Swedish registry

Introduction

Healthcare quality registries are important tools for benchmarking and quality assurance, as well as sources for population-based research. Studies utilizing population-based registries are essential for assessing outcomes following the implementation of trial results in the general population and in daily practice. The Swedish Rectal Cancer Register was launched in 1995 with the primary aim of monitoring outcomes after implementation of TME surgery and neoadjuvant radiotherapy. In 2007, colon cancer was included in the register, forming the Swedish Colorectal Cancer Register (SCRCR). To reflect the new era of multimodal treatment for colorectal cancer (CRC), the register was expanded to include preoperative staging, multidisciplinary tumor team (MDT) assessment, postoperative complications, and more operative variables.

All invasive colorectal adenocarcinomas (excluding autopsy findings) are registered, with data being prospectively entered by surgeons, radiologists, pathologists, and oncologists. Compared to the Swedish Cancer Register, with the exception of the first year, the coverage of registered cases has been stable between 96.0% and 99.6% for colon cancer and 98.6–100% for rectal cancer, respectively. Registration is performed electronically using a web-based platform. The recorded data include preoperative staging, operative treatment, histopathology, postoperative course, adjuvant therapy, and follow-up at 1, 3, and 5 years. Coding routines follow the national and international classification guidelines. Cancer staging and TNM classification follow the AJCC Cancer Staging Manual, 7th edition, and the TNM Classification of Malignant Tumors, UICC 7th edition. More than 300 published scientific papers have been based on data from SCRCR.¹

The SCRCR has in previous studies shown a good validity of the variables included,^2–6 the latest thorough validation was performed on records entered in 2008. The average completeness was 98.5%, timeliness was 98%, and comparability estimated to be comparable and reproducible with other registries. The validity, assessed by comparing re-abstracted data to original SRCRC data, showed an average agreement of 90%.⁶ Follow-up data have not been previously validated, registration forms have been updated, and new variables have been introduced since the last validation, such as the grading of postoperative complications according to the Clavien–Dindo (CD) classification.⁷ Thus, the selected new variables need to be further validated.

Hence, this study aimed to evaluate the validity of histopathological data, postoperative complications, and the 3-year follow-up data on cancer recurrence.

Materials and Methods

Of the 5173 patients who were reported to the SCRCR in 2015, 700 were randomly selected for this study. To ensure that the validation represented all three categories of hospitals in Sweden, a two-stage cluster sampling plan was used to randomly select patients from 12 of the 47 hospitals performing CRC surgery, proportionally distributed across all three hospital categories (university, regional, and sub-regional) and regions. The exclusion criteria were tumors other than invasive adenocarcinoma and metastatic disease at the time of diagnosis.

Medical files were reviewed for data re-abstraction of all pathologies, postoperative complications, and cancer recurrence variables, constituting 72 validated variables. Based on Bray,⁸ validity was defined as the proportion of cases in the SRCRC dataset in agreement with re-abstracted data. Re-abstraction was performed during the first half of 2023 by two surgeons blinded to the original data in the SCRCR who had prior experience in reporting data to the SCRCR. Registration forms, definitions, and coding instructions from 2015 were used in the re-abstraction process. The re-abstracted data were entered into a separate module on the web platform containing the selected variables only and later compared with the original data from the SCRCR. The agreement was subsequently calculated by direct comparison. For the date of recurrence, we accepted a maximum deviation of one week. Data accessed complies with relevant data protection and privacy regulations.

Statistics

Validity was defined as the percentage of exact agreement between the non-missing data in the SCRCR and the re-abstracted data. The strength of agreement was measured using Cohen’s kappa (κ) coefficient for categorical variables and Pearson´s correlation coefficient for numerical variables. The cut-off values according to Landis and Koch⁹ were used for qualitative interpretation of the kappa coefficients: κ = 0–0.20 (slight agreement); 0.21–0.40 (fair agreement); κ = 0.41–0.60 (moderate agreement); κ = 0.61–0.80 (substantial agreement); and κ = 0.81–1.0 (almost perfect agreement). In the case of five registered events or less, no kappa calculation was undertaken. Missing values in either the SCRCR or the re-abstracted data were excluded when calculating the kappa coefficient. Data management and statistical analyses were performed using the SPSS version 25 software (IBM SPSS, Armonk, NY, USA).

Ethics

This validation project was approved by the Swedish Ethical Review Authority (reference number 2021–02355). Informed consent was waived by the ethics committee, as all patients have consent on contributing with data to the registry to be used for research (after approval by the Swedish Ethical Review Authority). The study complies with the Declaration of Helsinki.

Results

Of the 700 patients selected, one was not available for validation and was therefore excluded from the study. Thus, 699 cases were validated and included in this study. Follow-up data were not available for three patients, as they had emigrated from the region and were thus excluded from the follow-up analysis. Seven cases were excluded from the histopathology validation because of the unavailability of pathology reports. The median proportion of missing data was, in total, 1.0% for the SCRCR and 2.1% for the re-abstracted data set, respectively. For pathology variables, missing data were 1.9% and 2.6%; for complication data, they were 0.0% and 1.5%; and for 3-year follow-up data, they were 0.0% and 2.1% in the SCRCR and re-abstracted data sets, respectively.

Histopathological Data

The median agreement between the 12 categorical histopathological variables was 93.4%, with a median kappa score of 0.83. T- and N-stages showed almost perfect agreement, 96.7% (κ = 0.95) and 97.2% (κ = 0.97), respectively (Table 1). Variables with substantial agreement according to kappa calculations were tumor grade (89.5%; κ = 0.73), Quirke´s TME assessment (91.0%; κ = 0.74), tumor deposit (89.2%; κ = 0.67). Microscopically radical resection showed a high exact agreement of 95.9% but substantial agreement according to kappa-calculation, κ = 0.43. The discrete variables of number of examined lymph nodes and number of positive lymph nodes showed a high agreement and high correlations, 98.2%, r = 0.94, and 94.9%, r = 0.97, respectively. Longitudinal and circumferential resection margins had an agreement of 79.9% (r = 0.87) and 88.2% (r = 0.82), respectively (Table 1 and Figure 1).

Table 1 Pathology Report Data. Comparison of Original Data in the Swedish Colorectal Cancer Registry and Re-Abstracted Data from Medical Files

	Missing Values in the SCRCR	Missing Values in Re-abstracted Data	Missing Values in Both Sources	Exact Agreement (Non-Missing Records) (%)	Cohen´s Kappa Score (κ)	Pearson´s Coefficient (r)
Quirke TME classification	98/216	76/216	58/216	91.0	0.74
T-stage	4/692	4/692	1/692	96.7	0.95
T1-stage	9/56	6/56	4/56	91.1	0.85
T3-stage	26/346	0/346	0/346	97.8	0.95
T4-stage	6/132	0/132	0/132	90.0	0.80
N-stage	6/692	12/692	3/692	96.3	0.93
Number of examined lymph nodes	9/692	14/692	5/692	98.2		0.94
Number of positive lymph nodes	14/692	17/692	6/692	94.8		0.97
Microscopically radical resection	7/692	24/692	5/692	95.9	0.43
Mucinous tumor	3/692	23/692	1/692	92.5	0.75
Perineural invasion	4/692	12/692	1/692	95.1	0.86
Vascular invasion	12/692	24/692	11/692	94.4	0.87
Tumor grade	2/692	15/692	0/692	89.5	0.73
LRM*	52/692	37/692	24/692	88.2		0.87
CRM**	54/692	42/692	33/692	85.9		0.82
Tumor deposit	8/692	18/692	1/692	89.2	0.67
- Number	17/80	10/80	5/80	96.6		0.97

Abbreviations: *LRM, Longitudinal resection margin; **CRM, Circumferential resection margin.

Figure 1 (A–E) Correlations of non-categorical pathology data.

Post-Operative Complication Data

This dataset consists of 44 variables, including five complication types and 17 subtypes, and their CD grading, where appropriate. The main variable postoperative complication had an exact agreement of 84.3% (κ = 0.61). The variables for specific types of complications exhibited low agreement, although surgical complications had a median agreement of 82.0% (κ = 0.63). Non-surgical complication variables showed lower agreement as in the case of postoperative infections and cardiovascular complications, which had only fair agreement (72.9% and 92.3% and κ = 0.24 and 0.39), respectively (Table 2). More common complications such as intra-abdominal infection (47 registered events in medical files and SCRCR combined) showed moderate agreement (81.1%, κ = 0.51), whereas anastomotic leak (32 events) showed substantial agreement (90.1%, κ = 0.71) (Supplementary Table 1). Many other subtype variables had few or no registered events, making exact agreement and kappa calculations irrelevant, and were thus presented as crosstabulations (Supplementary Table 2). The agreement on severe complications (CD 3b–5) was substantial (93.0%; κ = 0.75) in contrast to non-severe complications (CD 2–3a), which showed a fair agreement of 59.7% (κ = 0.32, Supplementary Table 3). Reoperation had an almost perfect agreement of 96.7% (κ = 0.87), whereas ICU treatment had a substantial agreement of 97.4% (κ = 0.76). The non-categorical data showed high correlations (Figure 2A–C).

Table 2 Postoperative Complication Data. Comparison of Original Data in the Swedish Colorectal Cancer Registry and Re-Abstracted Data from Medical Files

	Total Number of Events in Both Datasets	Missing Values in the SCRCR	Missing Values in Re-Abstracted Data	Missing Values in Both Sources	Exact Agreement (Non-Missing Records) (%)	Cohen´s Kappa score (κ)
Postoperative complications	273	0/699	11/699	0/699	84.3	0.61
Postoperative infection	98	N/A	N/A	N/A	72.9	0.24
Cardiovascular complications	29	N/A	N/A	N/A	92.3	0.39
Neurological complications	4	N/A	N/A	N/A	98.5
Surgical complications	132	N/A	N/A	N/A	82.1	0.63
Other complications	110	N/A	N/A	N/A	69.2	0.18
Intensive care (ICU)	23	0/699	1/699	0/699	97.4	0.76
Reoperation	38	0/699	1/699	0/699	96.7	0.87
Unplanned readmission	64	2/699	7/699	0/699	93.9	0.79

Abbreviation: N/A, Not applicable.

Figure 2 (A–C) Correlations of non-categorical postoperative data.

Three Year Follow Up

The variable cancer recurrence showed an agreement of 95.7% and a kappa coefficient of 0.86. The distinction between local and distant recurrence as first recurrence showed agreement of 85.9% and 89.1%, respectively, with moderate and fair kappa coefficients of 0.58 and 0.34, respectively (Table 3 and Supplementary Table 4). Dates of local and distant recurrences had an agreement of 62.8% and 61.5%, respectively, allowing for a week deviation but with good correlation (r = 0.9) (Figure 3A and B). The follow-up variables on MDT meeting (Figure 3C), treatment intention, and free of cancer at the time of follow-up showed moderate agreements, or 85.6% (κ = 0.51), 71.4% (κ = 0.48), and 80.1% (κ = 0.43), respectively.

Table 3 Three-Year Follow-Up Data. Comparison of Original Data in the Swedish Colorectal Cancer Registry and Re-Abstracted Data from Medical Files

	Missing Values in the SCRCR	Missing Values in Re-Abstracted Data	Missing Values in Both Sources	Exact Agreement (Non-Missing Records) (%)	Cohen´s Kappa Score (κ)
Cancer recurrence	1/521	5/521	0/521	95.7	0.86
Local recurrence	0/115	5/115	0/115	85.9	0.58
Distant metastasis	1/115	5/115	0/115	89.1	0.34
- Liver	N/A	N/A	N/A	89.7	0.79
- Lungs	N/A	N/A	N/A	91.0	0.82
- Other	N/A	N/A	N/A	67.9	0.26
MDT assessment	0/115	0/115	0/115	85.6	0.51
Treatment intention*	0/84	0/84	0/84	71.4	0.48
Free of cancer**	0/115	22/115	0/115	80.1	0.43

Notes: *The intention of treatment (curative, palliative or conversion) stated at MDT. **No sign of cancer at the 3-year follow-up.

Abbreviations: N/A, Not applicable; MDT, Multidisciplinary team assessment.

Figure 3 (A–C) Correlations of non-categorical follow-up data.

Discussion

This validation of the Swedish Colorectal Cancer Register showed good agreement, in general, between source data retrieved from medical files and register data in relation to pathology data and cancer recurrence, with the corresponding kappa values classified as almost perfect. As in several other studies, agreements on postoperative complication data were not as good.^3,10,11 Although there were generally few missing data in the register, there were some exceptions, particularly for some of the pathology variables in the validated dataset.

Some variables had a high percentage agreement but low kappa values. However, kappa calculations are affected by the prevalence of the events under consideration. Thus, in rare events, very low kappa values may not necessarily reflect low rates of overall agreement.¹² For instance, microscopically radical resection had an agreement of 95.9% but a kappa of 0.43. In a cross-tabulation, the SCRCR registrar and validator agreed on “yes” in 682 cases, “no” in 11 cases, and disagreed in 27 cases. In such cases, one can argue that the percentage of exact agreement provides a better perception of agreement than the strength of the kappa value.

The validity of the SCRCR data is comparable to that of other Scandinavian cancer registries.^13–16 The pathology data, in particular, were highly accurate, with the majority showing agreement of over 90% and kappa coefficients within the “close to perfect” range. These findings are important because many studies are based on tumor stages. The frequency of missing data was higher in the re-abstracted data than in SCRCR. This is probably due, at least in part, to shortcomings in the documentation of the medical files. The rate of missing values was the highest for Quirke TME assessment, which was incorporated into the register in 2014, the year before the year of this validation. It is likely that the assessment and reporting of the mesorectal fascia in pathology reports had not become an established routine for all pathologists at the time.

Data on postoperative complications showed high agreement regarding severe complications (CD 3b–5), but less agreement for non-severe complications (CD 2–3a). The strength of the agreement for complication type was poor for rare events, such as neurological and cardiovascular complications. One contributing factor is that, owing to the few events, a single mistake or misclassification leads to a high proportion of non-agreement. Moreover, Cohen´s kappa coefficient is affected by the number of categories of a variable, especially if the categories are binomial¹⁵ as in the majority of the validated variables. In addition, kappa is less reliable in cases of skewness in the dataset, such as when most cases fall into one category, which is a common situation in subgroup classifications. These factors must be considered, and some of the presented kappa values should be interpreted with caution.

The low agreement in the type and subtypes of complications may be due to different interpretations between the registrar and validator, as the delineation of the complication is not always straightforward. This was exemplified by the 41 patients registered with surgical complication by the validator, of whom 21 were registered as postoperative infection or other complications in the SRCRC. For instance, deep infectious complications may be registered as anastomotic leaks or sepsis, although better instructions and definitions have been introduced into the register over time. In the case of anastomotic leak, the agreement of the CD grade was low (65%, κ = 0.51) mainly because of differences in CD 3b and 4a scoring between registrar and validator. In general, the agreement of CD scores was weakest in low-grade complications (CD 2–3a), with an exact agreement of 62.5% and κ = 0.32, compared to strong agreement for higher grades (CD 3b–5) showing an exact agreement of 93.1% and κ = 0.75. This indicates that even if the CD classification system has been proven to have high reproducibility, the grading of complications can be the subject of opinion. Moreover, a moderate number of mild complications were not reported, as they were neglected in cases of more severe complications. This was reflected in the fact that the discrepancy was mainly noted for grade-2 complications with concomitant more severe complications. One reason for the low agreement in the postoperative complication dataset might, in part, be due to missing information in the validation process because of more restricted data security rules, such as the GDPR, as we did not have access to all medical files. Furthermore, some hospitals only provided paper copies upon request, with the consequent risk of missing unexpected events treated in other departments. However, underreporting in the register might also contribute, as it has been reported that the incidence of postoperative complications is higher in retrospective recordings than in prospective recording.¹⁷ This finding is supported by De la Rosette et al, who reported a lower rate of agreement in the grading of minor complications than in severe complications.¹⁸ These reports are consistent with the findings of the current study that inter-rater reliability is higher for more severe complications.

Overall, the validity of the recurrence data was good, although the specification of the site of recurrence, local or distant, showed discrepancies between data sources. The reason for this is unclear, but one contributing factor could be that only one site was registered, even though the patient had both local and distant metastases. However, the subgroup variables of liver and lung metastases showed strong agreement. Moreover, the agreement on the treatment intention for recurrence showed an agreement of only 71%. We hypothesize that this is mainly because at times this information was difficult to find in the provided medical files, as MDT notes can sometimes be found in the oncology files or in notes made after re-evaluation. A measure that would probably lead to improved validity for many variables would be the direct linking of the register to electronic data files. However, this requires a profound standardization of medical journals and implies data security and medicolegal issues. Although some attempts have been made, this approach has not yet been established in Sweden. Meanwhile, measures as more clear definitions of variables and better monitoring in the clinics are warranted.

A limitation of this study is that, for practical reasons, the re-abstraction of a single case was performed by only one validator and not by two independent validators, thus carrying an increased risk of errors. Another limitation is that data access was, to some degree, limited for security/integrity reasons, leading to the risk of some data being missed during re-abstraction. This risk was perceived to be higher for follow-up data, with a subsequent risk of underestimation of recurrence and the type of recurrence. Nevertheless, the strength of this validation was that a large study cohort randomly selected from all different hospital categories in Sweden based on a 2-step cluster procedure, taking size, location, and category into consideration. Another strength is that the validation was completely independent of the registered data but in an identical web-based manner.

In conclusion, this validation of SCRCR data showed high validity of pathology data and recurrence rates, whereas detailed data on recurrence were not as valid. Data on postoperative complications were less secure, although the incidence and CD grading of severe complications (grade 3b or more) were reliable.

Disclosure

The authors report no conflicts of interest in this work.

Acknowledgments

This paper is based on the thesis of Örvar Arnarson. It has been published on the institutional website: https://portal.research.lu.se/en/publications/emergent-colon-cancer-and-postoperative-complications-risks-and-m.

Additional information

Funding

This study was supported by Skane University Hospital grants, ALF-agreement (2018-Project 0264) between the Swedish government and the county councils, and local grant (Allmanna Sjukhusets i Malmö Stiftelse för bekampande av cancer).