Cognitive bias and the order of examination in forensic anthropological non-metric methods: a pilot study

ABSTRACT

Research has established that contextual information has the potential to influence the decisions of examiners in various forensic domains, including forensic anthropology. Studies have demonstrated that the non-metric sex estimation methods are susceptible to issues of cognitive bias, however the different types of stimuli that can influence the decision-making process remain understudied. As forensic anthropologists will examine multiple skeletal elements to estimate the sex of skeletal remains, a pilot study was designed to assess the potential of cognitive bias resulting from the order of examination. Two groups performed a non-metric sex estimation of the innominates and the skull with methods from Standards on one individual complete skeleton. Group A examined the skull first followed by the innominates, while Group B examined the innominates first followed by the skull. Results reveal a significant difference between the two groups in the sex estimation of the innominates and the complete skeleton (p = 0.020 and p = 0.022, respectively). This research demonstrates that order of examination for sex estimation may act as context and potentially influence the subsequent analyses. Additional research is necessary to broaden the understanding of decision-making and aid in the establishment of standard operating procedures designed to mitigate the potential effects of cognitive bias.

KEYWORDS:

• Forensic anthropology
• biological profile
• sex estimation
• cognitive bias
• decision-making
• visual assessments

Introduction

The decision-making process is an intrinsic aspect of forensic science ¹ and wherever there is decision-making and human interpretation there is also the potential for cognitive bias ^2–5. The perception and analysis of forensic evidence is arguably shaped by the experiences, prior beliefs, expectations, and emotions of the examiner, as well as task irrelevant contextual information, all of which can potentially result in decisions influenced by cognitive bias ^6,7. Empirical research has previously demonstrated that contextual information can influence the decisions of forensic examiners in various domains, including DNA analysis ⁸, fingerprint analysis ^9–11, crime scene investigation ^12,13, and forensic pathology ¹⁴, among others ^15–19, and forensic anthropology is no exception ^20–23. The medicolegal system relies on forensic science and therefore the potential of cognitive bias to affect the decision-making process should be minimized ²⁴. The field of forensic anthropology, as well as the broader forensic science community and governmental advisors and agencies, is committed to improving the understanding of the role of decision-making within forensic science and mitigating the potential of cognitive bias through empirical research, reports, and the introduction of standard operating procedures ^6–34.

However, it has been argued that some methods in forensic science may be at an increased risk of cognitive bias due to the level of subjectivity inherent within the technique ²⁵. For example, many of the forensic methods employed within the identification fields (e.g. finger mark comparison, bitemark evidence, morphological hair analysis) are based upon ‘visual comparison tasks’. Some of these methods rely on human interpretation, making them arguably more subjective and potentially more susceptible to issues of cognitive bias ³². Nevertheless, regardless of domain, methods employing visual comparison tasks are recognized in forensic science; however, it has been suggested that these types of subjective techniques would benefit from standardized operating procedures and clearly specified analytical protocols to reduce the potential of cognitive bias in the decision-making process ^32–34. Moreover, it has been argued that forensic examiners are at an increased risk of cognitive bias when dealing with evidence that is ambiguous ³⁵. When evidence is irrefutable, it is harder to rationalize a result to the contrary ³⁶, but when evidence is ambiguous, the examiner is arguably more at risk to be subconsciously influenced by task irrelevant information ^26,37.

Within forensic anthropology, the traditional sex and age estimation methods are based on visual assessments ²⁸. The most frequently employed sex estimation techniques are the non-metric methods of the pelvis and skull listed in Standards ^38,39. These methods provide a quick and reliable assessment, with accuracy levels reported to range from 84% to 96% ^40,41. However, these methods arguably possess a higher susceptibility to the potential of cognitive bias, due to their increased reliance on subjective human interpretation as well as the experience level of the observer ^42,43. Research has demonstrated that exposure to contextual information has the potential to influence the decisions of examiners regarding sex estimation ^21,22, and an additional study demonstrated that the exposure to an entire skeletal element has the potential to influence the decisions of examiners regarding the individual traits of that skeletal element ²⁰. Despite this research, the full extent of the different types of stimuli that have the potential to influence the decisions of forensic anthropologists remains unknown.

Forensic anthropologists perform multiple, successive examinations on different skeletal elements to estimate the sex on a set of human remains ⁴⁴. Estimating the sex of one skeletal element creates the opportunity for this decision to influence the subsequent decisions regarding additional skeletal elements, which in turn could influence the final sex estimation of the remains ⁴⁵. This principle has been demonstrated in Nakhaeizadeh et al. ²², where visual exposure to crime scene information influenced the subsequent decisions regarding sex estimation. It is important to identify where potential issues exist through empirical research and experimental studies in order to better understand the decision-making and human interpretation processes, as this will help to ensure the integrity and reliability of forensic science evidence ². Therefore, the aim of this experimental pilot study was to explore the potential of cognitive bias resulting from the order of examination during sex estimation of skeletal remains. This research investigated how the initial exposure to one skeletal element can influence and alter: 1) examiner decisions regarding the subsequent skeletal element; and 2) the final sex estimation of the remains, specifically when dealing with ambiguous traits.

Materials and methods

Research design

This experimental pilot study was developed to investigate how the initial exposure to and assessment of one skeletal element could potentially influence the subsequent assessment of a different skeletal element, as well as the final skeletal sex estimation when dealing with ambiguous traits. Two groups of participants (masters and PhD candidates studying forensic anthropology) were asked to perform a non-metric analysis of the skull and innominates of one complete skeleton to provide sex and age estimations. Group A examined the skull first followed by the innominates, while Group B examined the innominates first followed by the skull. Estimation of age-at-death was completed after the sex estimation was finished, in an effort to provide a more realistic examination for the participants.

Participants were not informed that there were two groups or given the full extent of the study at the beginning of their examination, as doing so may have affected their performance and results. Instead, participants were initially informed that the study was analysing methodological issues in non-metric sex and age estimation techniques, and were further informed of the full nature of the study after completion of all assessments. Additionally, participants were informed that the elements they were analysing came from a single individual skeleton. This study received approval from the Institute of Archaeology Ethics Committee, University College London (reference: 2017-18:033).

Materials

The single skeleton examined in this research was interred in a tomb in a post medieval cemetery (mid 1800s) and displayed undamaged sexually dimorphic features utilized to estimate sex. Associated with this tomb is a certificate of death from the General Register Office of Chichester, England, which indicates the remains to be male, and the initial osteological report associated with the collection also stated the remains to be male. However, no DNA analysis was undertaken and therefore the knowledge of ‘ground truth’ of the sex of the skeleton was absent. Conversely, this was not pertinent to the parameters of this research, as this research was not a methodological validation study.

These remains were selected because the sexually dimorphic features of the skull displayed some level of ambiguity, which can result in an increase in the influence of contextual information on decisions ^26,42,46,47. Additionally, a pre-experimental evaluation of the material included an osteological assessment from four different, independent, and experienced observers, all of whom noted while that the innominates displayed robust, male-indicating features, the skull displayed more gracile, indeterminate to female-indicating features. This evaluation was conducted in order to peer-review the trait observation scores of the remains from the osteological report in order to assure that the element of ambiguity was present.

By analysing a well-defined skeletal element first, either clearly robust or clearly gracile, an examiner will most likely form a first impression regarding the sex of that skeletal element. This prior analysis and early decision may act as contextual information, providing an expectation of the sex of a subsequent skeletal element. Therefore, this expectation has the potential to influence the subsequent analysis, particularly when working with ambiguous skeletal material.

This study formed two hypotheses. First, it was hypothesized that the initial examination of the innominates would influence the subsequent examination of the skull, as the pelvis is known to be the best indicator of sex ⁴⁸. Second, it was hypothesized that the order of examination would influence the participants’ decisions and alter the final sex estimation of the skeletal remains.

Participants

Contextual information has the potential to influence the decisions of both novices and experts in a similar manner ¹³. As such, the participants in this pilot study were graduate students within forensic and biological anthropology. Participants (n = 30) were divided into two groups of 15 (Table 1). Twenty-four of the 30 participants were enrolled in a taught master’s degree programmewhile the remaining six participants were doctoral candidates, with all participants studying bioarchaeology, forensic anthropology, or both. All participants were trained in the examination of skeletal remains, including the sex and age estimation methods employed in this study. Twenty-nine of the thirty participants received identical training as they were (or had previously been) enrolled on the same master’s programme. All participants were capable of performing a skeletal analysis that included non-metric sex and age estimation. All participants were voluntarily recruited and provided informed consent.

Table 1. Summary of participants.

	Group A	Group B
Gender
Female	12	13
Male	3	2
Education
MSc	12	12
PhD	3	3

Methods

The non-metric methods presented in Standards were selected for employment in this study as they are the most utilized techniques by biological anthropologists ^38,39. In addition to the sex estimation methods listed in Standards, the Genovés 1959 method reviewed by Bruzek ⁴⁹ was included. In an effort to more realistically mimic a forensic examination, participants were asked to perform an age assessment following the sex assessment. Participants employed the methods of Lovejoy et al. ⁵⁰ and Suchey-Brooks ⁵¹. Although age estimation data was collected, analysis of this data was not completed herein, as this falls outside of the remit of this study.

To estimate the sex of the skull, participants were instructed to use the method of Ascádi and Nemeskeri (1970) presented in Standards ³⁹, which observed the following traits: nuchal crest, mastoid processes, supraorbital margin, glabella, and mental eminence. To estimate the sex of the innominates, participants were instructed to use the methods of Buikstra and Ubelaker ³⁹, Phenice (cited in Bruzek ⁴⁹), and Genovés (cited in Bruzek ⁴⁹), which observed the following traits: shape of the pubis, ventral arc, subpubic concavity, ischiopubic ramus ridge, composite arch, and greater sciatic notch. Participants were instructed to score all traits individually first before estimating the sex of the skull and innominates. After estimating the sex of the skull and the innominates individually, participants were instructed to make a final sex estimation of the entirety of the skeletal remains.

Procedure

The skeletal remains were arranged in a supine anatomical position on a table prior to the arrival of any participants. The skull and the innominates (Figures 1–3) were each individually covered by a black cloth (Figure 4) to avoid any first impression influences, and a large black cloth was placed over the entire skeleton. All participants were provided with recording forms as well as a method reference packet. To maintain consistency, all participants were read the same script at the start of their analysis and were requested to follow the order of examination stated on their recording forms. The forms provided to Group A were arranged with the skull sex estimation sheet before the innominates sex estimation sheet, while the forms provided to group B were arranged with the innominates sex estimation sheet before the skull sex estimation sheet. The controlled order of examination for each participant group was important in order to assess the influence of the order of examination on the sex estimation of the remains. The age estimation sheet was last for both groups. Upon completion of the introductory script, the large black cloth was removed by the invigilator. This exposed the two smaller black cloths which covered the skull and innominates, but revealed several other skeletal elements. Participants were asked to remove the small black cloths in the order that they were examining the skeleton as the skeletal analysis progressed. This allowed participants to focus solely on the first skeletal element they were examining and prevented them from receiving influence from the subsequently examined skeletal element. Once a cloth was removed, it was set aside and the skeletal element was not covered again during the examination.

Figure 1. A left lateral view of the cranium to be examined by participants.

Figure 2. A left lateral view of the mandible to be examined by participants.

Figure 3. A ventral view of the left innominate to be examined by participants.

Figure 4. The skeletal remains to be examined by participants. Two small black sheets covered the skull and the innominates. A large black sheet was placed on top of this prior to participants entering the room and would be removed by the invigilator prior to the start of the examination.

To estimate the time needed for participants to assess the skeletal remains, a preliminary examination was run with several students who would not be taking part in this research. Based on these preliminary examinations, each participant was allotted 15 minutes to assess the skeletal remains. Additionally, each participant performed their examination alone.

Statistical analysis

The goal of the analysis was to establish if the two groups came to significantly different decisions as a result of the difference in the order of examination. To determine this, a series of Pearson chi-square followed by Fisher’s exact tests between the two groups were run at a 95% confidence interval using IBM SPSS Statistics v25 ⁵². The participants recorded their decisions in five sex estimation categories: female, possibly female, indeterminate, possibly male, and male. However, due to the small sample size of participants and low counts in certain recorded categories, the statistical analysis was performed on three compressed sex estimation categories: female/possibly female, indeterminate, and possibly male/male. This categorical data was converted into numerical coding in order to run the statistical analysis on SPSS.

Results

Participants’ decisions

The participants’ decisions of Group A and Group B are displayed in Tables 2 and 3, respectively.

Table 2. Group A participant decisions in three sex categories; Group A examined the skull first.

	Participants
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
Skull Sex Estimation	F	F	I	F	F	F	F	M	F	F	I	F	F	F	F
Innominate Sex Estimation	F	M	M	F	F	F	F	M	M	F	M	F	I	F	I
Complete Skeleton	F	M	I	F	F	F	F	M	M	F	M	F	F	F	I

F = Female/Possibly Female, I = Indeterminate, M = Male/Possibly Male.

Table 3. Group B participant decisions in three sex categories; Group B examined the innominates first.

	Participants
	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30
Innominate Sex Estimation	M	F	M	M	M	M	M	I	I	M	M	I	I	M	I
Skull Sex Estimation	M	F	M	M	F	F	F	F	F	F	I	F	F	I	F
Complete Skeleton	M	F	M	M	I	I	I	F	I	M	M	I	I	M	I

F = Female/Possibly Female, I = Indeterminate, M = Male/Possibly Male.

Chi-square test between Group A and Group B: skull assessment

Decisions regarding the sex of the skull (Figure 5) showed the majority of participants in Group A and Group B estimated the skull to be female/possibly female, at 80% and 66.7% respectively. The chi-square test (Table 4) indicated there was no significant difference in the sex estimation of the skull between the two groups (p = 0.554). See Table 4 for a full report on the chi-square and Fisher’s exact tests.

Table 4. Chi-square test results: skull assessment.

	Value	df	Asymp. Sig.	Exact Sig.
Pearson Chi-Square	1.182	2	0.554
Fisher’s Exact Test	1.239			0.836

Figure 5. The distribution of the sex estimation of the skull in three categories. Group A examined the skull first, while Group B examined the innominates first.

Chi-square test between Group A and Group B: innominate assessment

Decisions regarding the sex of the innominates (Figure 6) showed 53.3% of Group A estimated the innominates to be female/possibly female, while 60% of Group B estimated the innominates to be male/possibly male. The chi-square test (Table 5) indicated that there was a significant difference in the sex estimation of the innominates between the two groups (p = 0.020). See Table 5 for a full report on the chi-square and Fisher’s exact tests.

Table 5. Chi-square test results: innominate assessment.

	Value	df	Asymp. Sig.	Exact Sig.
Pearson Chi-Square	7.873	2	0.020
Fisher’s Exact Test	7.748			0.027

Figure 6. The distribution of the sex estimation of the innominates in three categories. Group A examined the skull first, while Group B examined the innominates first.

Chi-square test between Group A and Group B: final sex estimation assessment

Decisions regarding the final sex estimation (Figure 7) showed 60% of Group A estimated the skeletal remains to be female/possibly female, while 60% of Group B estimated the skeletal remains to be indeterminate. The chi-square test (Table 6) indicated that there was a significant difference in the final sex estimation of the skeletal remains between the two groups (p = 0.022). See Table 6 for a full report on the chi-square and Fisher’s exact tests.

Table 6. Chi-square test results: final sex estimation assessment.

	Value	df	Asymp. Sig.	Exact Sig.
Pearson Chi-Square	7.632	2	0.022
Fisher’s Exact Test	7.444			0.029

Figure 7. The distribution of the final sex estimation of the skeletal remains in three categories. Group A examined the skull first, while Group B examined the innominates first.

Discussion

This pilot study aimed to investigate the potential of cognitive bias in the non-metric sex estimation methods resulting from the order of examination when dealing with ambiguous traits. As forensic anthropologists will perform multiple examinations on different skeletal elements during their analysis to estimate sex ⁴⁴, this research was designed to explore how the initial exposure to one skeletal element can influence and alter: 1) observers’ decisions regarding the subsequent skeletal element, and 2) the final sex estimation of the remains. The study focused specifically on non-metric sex estimation methods to test for this effect further. The statistical analysis revealed there was no significant difference in the decisions between the two groups regarding the sex estimation of the skull. However, the analysis revealed there was a significant difference between the two groups regarding both the sex estimation of the innominates and the final sex estimation of the remains.

Regarding the first aim of this pilot study, it was hypothesized that the initial assessment of the innominates would influence the subsequent assessment of the skull, as the innominates are known as the most reliable skeletal element for sex estimation and displayed strongly male-indicating features. However, the results revealed no significant difference in the sex estimation of the skull, p = 0.554. This indicates that the initial assessment and sex estimation of the innominates did not influence the subsequent assessment and sex estimation of the skull. Additionally, while it was not hypothesized that the initial assessment of the skull would influence the subsequent assessment of the innominates, the results revealed there was a significant difference in the sex estimation of the innominates, p = 0.020. This indicates that the initial assessment and sex estimation of the skull did influence the subsequent assessment and sex estimation of the innominates.

While the hypothesis was not supported by the findings, this experimental pilot study did demonstrate that initial exposure to one skeletal element can potentially alter examiner decisions regarding a subsequent skeletal element. When a forensic anthropologist is examining one skeleton, there is potentially an expectation that the different skeletal elements analysed will all indicate the same sex. For example, if an individual set of skeletal remains were estimated to be female, an examiner may subconsciously accept that both the innominates and the skull would display traits that indicate female. However, this is only representative of an ideal situation. In reality, there is considerable overlap between in size and shape between males and females, resulting in individuals that may present with both male-indicating and female-indicating traits on either the same or different skeletal elements ^53,54.

In this study, it appears that the initial examination of the skull created an expectation of the sex of the skeletal remains. The 15 participants in Group A examined the skull first, and 12 out of these 15 participants (80%) estimated the skull to be female/possibly female. Of the 12 participants that estimated the skull to be female/possibly female, eight of them (67%) also estimated the innominates to be female/possibly female. Examining the skull first could have created a subconscious expectation of the sex of the rest of the skeleton, and it was this expectation that potentially influenced the subsequent examination of the innominates. Participants in Group A went on to examine the innominates in a manner that aligned with the previously estimated sex of the skull, therefore confirming their expectations. Arguably, the initial examination and decision regarding the sex of the skull influenced subsequent interpretation of the innominates. However, while the sex of the skull is relevant information when estimating the sex of skeletal remains as a whole, it is not relevant to the individual analyses of other skeletal elements. When examining multiple skeletal elements to estimate sex, each element must be analysed independently and free of influence from other elements.

Regarding the second aim of this pilot study, it was hypothesized that the order of examination would influence the participants’ decisions and alter the final sex estimation of the remains. The results revealed a difference in the sex estimation of the remains between the two groups, p = 0.022. As previously mentioned, the 15 participants in Group A examined the skull first, and 12 out of these 15 participants estimated the skull to be female/possibly female. Of the 12 participants that estimated the skull to be female, nine of them (75%) also estimated the complete skeleton to be female/possibly female. Additionally, overall in Group A, 11 out of 15 participants (73%) made a final decision regarding the sex of the complete skeleton that matched their original decision regarding the initial examination and sex estimation of the skull.

Similar results were observed in Group B. The fifteen participants in Group B examined the innominates first and nine out of these 15 estimated the innominates to be male/possibly male. Of the nine participants that estimated the innominates to be male/possibly male, six of them (67%) also estimated the complete skeleton to be male/possibly male. Additionally, overall in Group B, 11 out of 15 participants (73%) made a final decision regarding the sex of the complete skeleton that matched their original decision regarding the initial examination and sex estimation of the innominates. Moreover, regardless of order of examination and first skeletal element examined, 22 out of the 30 total participants (73%) matched their final sex estimation of the complete skeleton with the sex estimation of the initial element examined. This, coupled with the difference in the participants’ decisions between the two groups regarding the final sex estimation, indicates that the order of examination did influence the sex estimation of the skeletal remains as a whole. Across both groups, participants had the tendency to make a final sex estimation that agreed with their initial examination and hypothesis.

This experimental pilot study in non-metric sex estimations identified that when dealing with ambiguous skeletal remains, the order of examination has the potential to influence participants’ decisions regarding both subsequent skeletal elements as well as the remains as a whole. Additional research is therefore necessary to further explore the possibility of influence resulting from the order of examination. In order to create procedures designed to minimize the potential of cognitive bias and maintain the reliability of forensic science evidence, the potential presence of cognitive bias must first be identified in all stages of the forensic process ^2,55. One approach to study the potential cognitive effects is through experimental research. As demonstrated by this study and others ^2,9–22,45, cognitive bias may come in different forms and has the potential to alter the reliability of methods and influence the decisions of examiners. By further analysing the potential of cognitive bias resulting from the order of examination as well as other variables, future studies will be able to better inform on procedures that can help to mitigate the effects of these biases.

Following similar studies in forensic anthropology, this research employed the participation of graduate students to examine the potential of cognitive bias in frequently employed methods in forensic anthropology ^21–23. Although all participants were trained and capable of performing sex estimates, the examination of the expertise of the participants was outside the scope of this research. Participants in this study had similar amounts of training, which was important to control the variable of knowing that they could perform the methods. Moreover, the literature has identified that experts are not immune from cognitive bias and the potential of contextual information to influence the decision-making process ^6,43,56. Empirical research into the potential of cognitive bias in other domains has demonstrated that experts are, in the least, affected as much as novices by contextual information (e.g. ¹⁴). To explore this possibility, further research using experts and more experienced observers is necessary to examine the effects of the order of non-metric sex estimation of skeletal remains and its potential effects on expertise. The data generated from this research provides a starting point for assessing how the order of examination in forensic anthropological methods may influence subsequent assessments and conclusions.

An additional limitation of this study is the small number of participants. By employing a cohort who received identical training from attending the same master’s programme, this study was able to control and account for the variable of education. However, this came as a trade-off and resulted in a limited number of participants meeting the inclusion criteria. Further research is needed to draw generalized conclusions across the forensic anthropological board with increased sample sizes and a range of experienced observers. In addition, it could potentially be argued that in forensic anthropological casework, a professional examiner would employ as many methods as possible including non-metric sex estimation methods ²⁸. Forensic anthropologists, with the knowledge that the pelvis is the most accurate indicator of sex, would consistently begin skeletal analyses with the examination of the pelvis, if present. However, it is important to highlight that not all cases might have all skeletal remains present, and sometimes examiners may not be able to perform both metric and non-metric sex estimations, due to various reasons. Experimental studies can therefore help control for and understand some of the potential effects that could be introduced in some of the procedures employed when dealing with limitations. Moreover, it is pertinent that both empirical research and professional experience be employed in the forensic science process ^1,57. In fact, recent governmental reports have established the need for forensic practitioners to not rely solely on experience, or use forensic casework as a substitute for empirical research ^25,32. As such, this experimental pilot study provides a starting point of empirical research to help support the aforementioned practices and their continued employment in forensic casework.

Undertaking research that builds on this approach and these findings will generate additional data that will aid in developing a greater understanding of which factors lead to and influence the interpretations reached in the assessment of skeletal remains ²⁸. Collectively, research can then begin to offer insight into standard operating procedures which would help to mitigate and minimize the potential of cognitive biases due to the order of examination.

Conclusion

This pilot study adds to the existing empirical research examining the potential of cognitive bias in forensic anthropology ^20–23. The aim of this research was to examine the potential of cognitive bias in non-metric sex estimation methods as a result of the order of examination, specifically when dealing with ambiguous traits. The results of this research demonstrate that the initial exposure to one skeletal element can influence and alter both the observers’ decisions regarding the subsequent skeletal element, as well as the final sex estimation of the remains. This study indicates that the non-metric sex estimation methods frequently employed in forensic anthropology are potentially susceptible to issues of cognitive influences due to the order of examination. Continued empirical research is necessary to further the understanding of the role of decision-making in the forensic anthropological process as well as aid in the creation of standard operating procedures designed to mitigate any potential of cognitive biasing effects.

Acknowledgments

The authors would like to thank Clara Gannon, Britny Martlin, and Conner Welty for their help and assistance in the trial run of this study. Additionally, the authors would like to extend their gratitude to all of the participants who donated their personal time to partake in this research.

Disclosure statement

No potential conflict of interest was reported by the author(s).