An inter-observer Ki67 reproducibility study applying two different assessment methods: on behalf of the Danish Scientific Committee of Pathology, Danish breast cancer cooperative group (DBCG)

Abstract

Introduction: In 2011, the St. Gallen Consensus Conference introduced the use of pathology to define the intrinsic breast cancer subtypes by application of immunohistochemical (IHC) surrogate markers ER, PR, HER2 and Ki67 with a specified Ki67 cutoff (>14%) for luminal B-like definition. Reports concerning impaired reproducibility of Ki67 estimation and threshold inconsistency led to the initiation of this quality assurance study (2013–2015). The aim of the study was to investigate inter-observer variation for Ki67 estimation in malignant breast tumors by two different quantification methods (assessment method and count method) including measure of agreement between methods.

Material and methods: Fourteen experienced breast pathologists from 12 pathology departments evaluated 118 slides from a consecutive series of malignant breast tumors. The staining interpretation was performed according to both the Danish and Swedish guidelines. Reproducibility was quantified by intra-class correlation coefficient (ICC) and Lights Kappa with dichotomization of observations at the larger than (>) 20% threshold. The agreement between observations by the two quantification methods was evaluated by Bland–Altman plot.

Results: For the fourteen raters the median ranged from 20% to 40% by the assessment method and from 22.5% to 36.5% by the count method. Light’s Kappa was 0.664 for observation by the assessment method and 0.649 by the count method. The ICC was 0.82 (95% CI: 0.77–0.86) by the assessment method vs. 0.84 (95% CI: 0.80–0.87) by the count method.

Conclusion: Although the study in general showed a moderate to good inter-observer agreement according to both ICC and Lights Kappa, still major discrepancies were identified in especially the mid-range of observations. Consequently, for now Ki67 estimation is not implemented in the DBCG treatment algorithm.

Introduction

The classification of malignant breast tumors into molecular intrinsic subtypes by DNA microarrays (luminal A, luminal B, HER2-enriched and basallike) were introduced by Perou et al. [1]. The study contributed with new insights into the molecular landscape of breast cancer providing important prognostic information concerning both disease free survival and overall survival for each of the molecular intrinsic subtypes with luminal B tumors having poorer outcome than luminal A tumors. The main difference between luminal A and luminal B is related to the higher expression of proliferation related genes in luminal B tumors [2] and it has been confirmed that genes indicative of high tumor cell proliferation are the major contributors of poor prognosis in various prognostic gene assays [3]. Since multigene testing is an expensive procedure not applicable in every pathology laboratory immunohistochemical staining (IHC) for the proliferation marker Ki67 has been an obvious choice. Ki67 is expressed in the cell nucleus during cell cycle in the G1, S, G2 and M phase but not in the G0 phase. Several studies have documented the prognostic and predictive value of Ki67 expression as a continuous variable in both the adjuvant and neoadjuvant setting [4–6]. Cheang et al. [7] showed that a clinical relevant ki67 IHC cut point could be determined from ROC curves to separate luminal A and luminal B tumors as compared to the PAM50 gene expression signature as gold standard. The initial training set consisting of 144 tumors identified a Ki67 threshold of 13.25% separating the luminal A-like and luminal B-like tumors in two distinct prognostic groups. Validation of this finding by IHC in a larger population of ER positive HER2 normal cases (N = 2276) confirmed the initial results with a cutoff value of 14%. The study was performed on tissue microarrays (TMA) and a population with excellent prognosis was identified indicating that postmenopausal patients with luminal A-like tumors might be spared chemotherapy. Encouraged by the results from this study the 2011 St. Gallen Consensus Conference [8] introduced the use of pathology to define the intrinsic breast cancer subtypes by application of IHC surrogate markers (ER, PR, HER2 and Ki67) with a specified Ki67 cutoff (>14%) for luminal B-like definition. The Ki67 cutoff value for luminal B-like classification was further changed to ≥20% at the 2013 St. Gallen Consensus Conference [9] based on the work of Prat et al [10]. In addition, the International Ki67 in Breast Cancer working group published recommendations for Ki67 assessment for both the pre-analytical, analytical and post-analytical phase emphasizing the need for standardization of procedures [11].

Since 2010, Ki67 IHC has been performed on all malignant breast tumors in Denmark. Due to concerns about the reproducibility of Ki67 interpretation and quantification the Danish Scientific Committee of Pathology in the DBCG initiated and finalized a Ki67 quality assurance study during a two-year period (2013–2015). Eleven out of twelve Danish departments of pathology involved in breast cancer diagnostics participated in the study in collaboration with Department of Pathology, Skåne University Hospital, Lund, Sweden.

The aim of the study was to investigate inter-observer variation for Ki67 estimation in malignant breast tumors by two different methods according to the national guidelines for Ki67 staining interpretation in Denmark and Sweden including measure of agreement between methods.

Material and methods

Material

Fourteen experienced breast pathologists evaluated 118 slides from a consecutive series of malignant breast tumors as part of this quality assurance study.

The Ki67 staining was performed centrally with CONFIRM anti-Ki67 Rabbit Monoclonal Primary Antibody 30-9 (Roche/Ventana a/s) according to standard procedure (http://www.nordiqc.org/). The slides were circulated among the observers and evaluated locally by standard light microscopy on full tumor sections. Five samples had missing observations by the assessment method and six samples by the count method resulting in a total number of 1647/1646 observations for the assessment method and count method, respectively.

Staining interpretation

Positive ki67 staining was defined as any brown stain in the nucleus above background.

The staining interpretation was performed according to national guidelines:

The Danish interpretation guideline recommends a semi-quantitative evaluation of Ki67 nuclear staining in hotspot areas with notation of the percentage of Ki67 positive invasive tumor cells in 5–10% intervals (assessment method).

The Swedish interpretation guideline recommends calculation of 200 invasive tumor cells in hotspot areas with notation of the number of Ki67 positive tumor cells in percentage (count method). Figure 1 represents an example of nuclear Ki67 staining with variable nuclear staining intensity.

Figure 1. Nuclear Ki67 immunohistochemical staining demonstrating variation in staining intensity.

Statistics

The statistical analysis was performed by the DBCG statisticians. The distribution of observations by the assessment and count methods is presented in histograms and inter-observer variability is visualized in box-plots. Intra-class correlation coefficient (ICC) was used as a summary measure of inter-observer reproducibility and for this purpose the two-way random effect model was applied [12]. The ICC has a range of 0–1, with one denoting the highest agreement. Since there are no standard values for acceptable reliability of ICC, it was decided that a prespecified value of ICC > 0.80 was to be considered indicative of good agreement in this study as compared to kappa statistics with 0.8–1.0 regarded as almost perfect [13–15].

Following dichotomization of observations at the larger than (>) 20% threshold the agreement between individual observers was calculated as the proportion of overall positive and negative agreement and as Light’s Kappa [16]. The Kappa statistics is not defined in case of missing observations; hence, the dataset was reduced to 113 samples evaluated by all observers and by both methods. The agreement between observations done by the assessment and count methods was evaluated in a Bland–Altman plot. The plot depicts the difference between observations of the same sample by the same observer against the mean of observations [17]. If the differences have constant mean and variance the limits of agreement (LoA) can readily be added to the plot. For the Ki67 observations the mean and the variance of differences were clearly not constant over the range of observations and for this reason variance stabilizing logit and arcsine transformations were evaluated. By arcsine transformation variance homogeneity was obtained, whereas a linear trend of differences remained. Therefore arcsine transformation was considered as a valid model to be used for predictions of the outcome of one method, by knowing the observation of the other method.

In case of transformed observations the Bland–Altman plot and associated LoA do not offer an easy visual interpretation. To make a clinically meaningful presentation of the prediction interval this was determined according to Carstensen et al. [18] and back-transformed to the original scale.

Results

Observations by the assessment method were in general skewed towards lower values compared to observations by the count method (Figure 2). The assessment method had a lower mean value of 35.3% (95% CI: 33.97–36.65) vs. 37.4% (95% CI: 36.12–38.67) for the count method but a larger dispersion: The standard deviation was 27.8 vs. 26.4 and distance between the 10%-percentile and the 90%-percentile was 75-percentage-points vs. 74-percentage-points. As can be seen from the confidence intervals the means of the two methods are not statistically significantly different.

Figure 2. Distribution of registrations by the assessment and count methods on the original scale as percent positively stained tumor cells.

For the fourteen raters the median ranged from 20% to 40% by the assessment method and from 22.5% to 36.5% by the count method (Figures 3(A,B)).

Figure 3. Distribution of Ki67 observations by the (A) assessment and (B) count methods according to rater. The bottom and top edges of the box are located at the sample 25th and 75th percentiles. The center horizontal line is drawn at the 50th percentile (median) and the circle illustrates the mean value.

The ICC was 0.82 (95% CI: 0.77–0.86) by the assessment method vs. 0.84 (95% CI: 0.80–0.87) by the count method.

For Ki67 observations dichotomized at the 20% threshold 57.4% were positive by the assessment method whereas 67.6% were positive by the count method. It is shown that 195 (28%) of the observations ≤20% by the assessment method was estimated as above 20% by the count method and 26 (3%) of the observations above 20% by the assessment method was estimated as ≤20% by the count method (Table 1). The proportion of between-method agreement of individual observers for observations above 20% ranged from 0.73 to 0.96 (median 0.87), and, as more observations were classified as positive by both methods, the proportion of positive between-method agreement (range 0.78–0.97, median 0.90) was larger than the proportion of between method agreement for observations ≤20% (0.65–0.91, median 0.83). Light’s Kappa was 0.66 for observation by the assessment method and 0.65 by the count method (n = 1582).

Table 1. Observations of Ki-67 by the assessment and count methods dichotomized at the >20% threshold*.

	Count method
Assessment method	≤20%	>20%	Total
≤20%	507	195	702
>20%	26	918	944
Total	533	1113	1646

* Proportion of overall agreement: (507 + 918)/(507 + 918 + 195 + 26) = 0.866.

Proportion of positive agreement: (2*918)/(2*918 + 195 + 26) = 0.893.

Proportion of negative agreement: (2*507)/(2*507 + 195 + 26) = 0.821.

A Bland–Altman plot (Figure 4(A)) visualized the agreement between the count and assessment methods. It is seen that the difference between observations is small at the lower and upper end of the scale but large in central part of the scale. At the low end of the scale observations by the count method tend to be larger than observations by the assessment method (as seen by the regression line), whereas the opposite tends to be the case in the high end of the scale. Due to this trend in differences and the absence of constant variance, the fitted LoA are not reliable. In Figure 4(B), the regression line and prediction limits are back-transformed from the arcsine-scale. The regression line in Figure 4(B) shows that at the 20% threshold for the assessment method the prediction of the count method is 23.4% (95% PI: 8.2–43.5%), vice versa at the 20% threshold for the count method the prediction of the assessment method is 16.7% (95% PI: 2.6–36.8%)

Figure 4. Ki67 observations done by the assessment and count methods (n = 1646). (A) Bland–Altman plot. The variance of the difference between observations is clearly not constant; hence the suggested limits of agreement (dashed lines) are not meaningful. For ease of interpretation overlapping observations are made visible by adding random noise. (B) Two-way prediction limits for observations. The regression line (solid line) and the 90% and 95% prediction interval (dashed lines) are back-transformed from the arcsine-scale. For ease of interpretation overlapping observations are made visible by adding random noise to values observed by the assessment method.

An observation of 5% by the assessment method has 95% chance of being Ki67 negative (≤20%) on the count scale, whereas an observation of 10% by the assessment method has 95% chance of being Ki67 negative on the assessment scale.

An observation of 33% by the assessment method has 95% chance of being Ki67 positive (>20%) on the count scale, whereas an observation of 40% by the assessment method has 95% chance of being Ki67 positive on the assessment scale.

Discussion

Despite of controversies concerning Ki67 standardization of interpretation and cutoff levels the Ki67 IHC labeling index is generally accepted as an important prognostic factor in ER positive breast cancer being the main discriminator in classification of luminal A-like and luminal B-like breast cancer. In this study the count method resulted in a higher proportion of cases with Ki67 > 20% as compared to the assessment method (Figure 2). Also, the study showed almost similar kappa values (0.66 for the assessment method and 0.65 for the count method) consistent with similar level of disagreement for both methods although the prespecified ICC level was achieved especially for the count method by 0.84 (95% CI: 0.80–0.87).

The magnitude of estimators of agreement (kappa or ICC) is conventionally interpreted as follows: 0 (absent), 0–0.19 (poor), 0.20–0.39 (weak), 0.30–0.59 (moderate), 0.60–0.79 (good), and ≥0.80 (almost complete agreement). This interpretation is however arbitrary without objective argumentation for the specified intervals. Consequently, despite of kappa values of 0.664 for the assessment method and 0.649 for the count method in the present study and as such indicating good inter-observer agreement according to conventional kappa interpretation, it must be reconsidered whether this level of agreement is acceptable when it relates to clinical treatment decision. An alternative approach for kappa interpretation has been suggested with kappa values above 0.8 as recommended prior to clinical implementation [19].

With respect to the measure of agreement between methods the Bland–Altman plot confirmed that the highest agreement in Ki67 observations in this study was in the very low and very high end of the scale with impaired agreement in the mid-range of observations (Figure 4(A,B)). This resulted in regrouping of a large number of observations depending on quantification method (Table 1) which is critical since a major part of Ki67 estimations in daily routine diagnostics are in the range of 10–30%. These findings in combination with the fact that the inter-observer agreement (kappa value and ICC) for both methods were almost identical lead to the conclusion that implementation of a prespecified Ki67 cutoff level was not advisable, thus in line with the 2017 St. Gallen Consensus Conference publication [20]. The presented results in this study are in accordance with those of others documenting impaired Ki67 inter-observer agreement [13,21–24]. Leung et al. [15] and Polley et al. [23] showed that calibration and standardization of scoring methods on TMA and core needle biopsies improved ICC to above 0.90 but still major discrepancies persisted around clinical important cutoff values. In addition, other studies have documented high level of inter-observer variability in the gray zone area of Ki67 index of 10–30% [21,24–26]. The concerns regarding interpretation of Ki67 are related to several contributing factors other than inter-observer reproducibility and lack of standardization of staining interpretation. Also inter-laboratory discrepancies with relation to IHC platform, choice of Ki67 antibody (clone) and IHC detection system are to be considered [22,27]. The Swedish survey by Ekholm et al. [27] reported good inter-laboratory reproducibility for Ki67 with central review. However, the Swedish laboratories had lab-specific thresholds for Ki67 and a lower agreement was reported between observers and central review when the lab-specific cutoff levels were used (κ = 0.57). In the recent paper by Focke et al. [22] on behalf of the German Breast Screening Pathology Initiative, thirty European pathology laboratories stained serial sectioned TMA slides according to local routine protocols. Central Ki67 assessment was performed reporting the proportion of tumors classified as luminal A-like after dichotomizing observations at the ≥14% threshold. The study showed a huge inter-laboratory variation in luminal A-like classification ranging from 17% to 57% (p < .0001).

The strength of this study is related to the fact that experienced breast pathologists performed the staining interpretation in accordance to the national guidelines in Denmark and Sweden.

Further, the staining procedure was done centrally and the staining interpretation was performed on full sections. There are however some potential limitations to be considered. Standard light microscopy was applied in the present study since automated image analysis is not standard procedure for Ki67 estimation in the Danish departments of pathology. Recent promising results regarding improvement of Ki67 reproducibility by computer assisted image analysis warrants further investigation of this method [28,29]. Also, this study did not include neither complete subtype classification by IHC surrogate markers nor validation by PAM50 gene expression as gold standard. Recent studies have demonstrated that classification of the intrinsic subtypes by molecular gene expression profiling is superior to classification by IHC surrogate markers [30,31].

Consequently, when implemented as part of the surrogate IHC panel for the intrinsic subtypes the documented inconsistency in Ki67 estimation in combination with the lack of agreement concerning Ki67 thresholds [8,9,20,32] might course either under- or over treatment on the individual patient level.

Based on the present study the Scientific Committee of Pathology in the DBCG concluded that for now Ki67 IHC index should not be introduced in the DBCG treatment algorithm. Due to the documented level of evidence (1B) the PAM50 multigene test was included in the national DBCG guidelines in 2017 and is presently offered to a subset of the Danish postmenopausal ER+, HER2 negative breast cancer patients (www.dbcg.dk) [33,34].

In conclusion, although the study in general showed good inter-observer agreement according to both ICC and Lights Kappa, still major discrepancies were identified in especially the mid-range of observations. The study confirmed the importance of standardization and validation of procedures prior to implementation of (bio) markers for treatment guidance in national guidelines (http://www.nordiqc.org/) [35–37].

Acknowledgments

The authors thank Stina Lyck Carstensen* and Karsten Bjerre for assistance with the statistical analyses.

*Danish Breast Cancer Group Secretariat, Rigshospitalet, Copenhagen, Denmark.

Disclosure statement

Maj-Lis Møller Talman and Tomasz Piotr Tabor have received honoraria from Roche education session. Other authors declare that the have no potential conflict of interest.