Algorithmic bias: review, synthesis, and future research directions

Figures & data

Figure 1. Stages of the article selection process.

Table 1. Confusion matrix for risk assessment and loan application decisions, adapted from Verma and Rubin (2018)

	Predicted: Default (Ŷ = 0)	Predicted: Not default (Ŷ = 1)	Bias metrics
Truth: Default (Y = 0)	True Negative (TN)	False Positive (FP)	True negative rate: TN/(TN + FP) False positive rate: FP/(TN + FP)
Truth: Not default (Y = 1)	False Negative (FN)	True Positive (TP)	True positive rate: TP/(TP + FN) False negative rate: FN/(TP + FN)
Bias metrics	Negative predictive value: TN/(TN + FN)	Positive predictive value: TP/(TP + FP)	Overall selection probability: (FP + TP)/(FP + TP + TN + FN) Overall accuracy rate: (TN + TP)/(FP + TP + TN + FN)

Table 2. A non-exhaustive list of equality objectives and corresponding metrics and constraints, adapted from Shrestha and Yang (2019) and Verma and Rubin (2018)

Equality objective	Description based on metrics	Constraint
Demographic parity	Selection probabilities should be equal between disadvantaged and advantaged groups.	Pr(Ŷ = 1 | A = 0) = Pr(Ŷ = 1 | A = 1)
Predictive parity	Positive predictive values should be equal between disadvantaged and advantaged groups.	Pr(Y = 1 | A = 0, Ŷ = 1) = Pr(Y = 1 | A = 1, Ŷ = 1)
Error rate balance	False positive rates and false negative rates should be equal between disadvantaged and advantaged groups.	Pr(Ŷ = 1 | A = 0, Y = 0) = Pr(Ŷ = 1 | A = 1, Y = 0), Pr(Ŷ = 0 | A = 0, Y = 1) = Pr(Ŷ = 0 | A = 1, Y = 1)
Equalised odds	True positive rates and false positive rates should be equal between disadvantaged and advantaged groups.	Pr(Ŷ = 1 | A = 0, Y = 1) = Pr(Ŷ = 1 | A = 1, Y = 1), Pr(Ŷ = 1 | A = 0, Y = 0) = Pr(Ŷ = 1 | A = 1, Y = 0)
Equal opportunity	True positive rates should be equal between disadvantaged and advantaged groups.	Pr(Ŷ = 1 | A = 0, Y = 1) = Pr(Ŷ = 1 | A = 1, Y = 1)

Note: Females: A = 1, Males: A = 0

Table 3. Derived, extrapolated, and new relations proposed in this study

Proposition	Conceptual and empirical reviewed papers that support the propositions	Derived or extrapolated relations from the reviewed papers that require further empirical verification	New relations proposed in this study that require empirical testing
P1	Lee et al. (2017); Saxena et al. (2019)	N/A	Algorithmic bias → Perceived fairness
P2	Ebrahimi and Hassanein (2019)	Perceived fairness → Behavioural responses (recommendation acceptance)	Perceived fairness → Behavioural responses (algorithm appreciation, system adoption)
P3	Rossi (2019); (Ebrahimi & Hassanein, 2019); (Dodge et al., 2019);	Propensity to trust in technology × Perceived fairness → Behavioural responses (system adoption) Cognitive style (need for cognition) × Algorithmic bias → Perceived fairness	Biased attitudes and beliefs × Algorithmic bias → Perceived fairness Moral identity × Algorithmic bias → Perceived fairness Moral identity × Perceived fairness → Behavioural responses Propensity to trust in technology × Algorithmic bias → Perceived fairness Propensity to trust in technology × Perceived fairness → Behavioural responses (recommendation acceptance, algorithm appreciation)
P4	M. Lee (2018a); Barlas et al. (2019); Ebrahimi and Hassanein (2019); (Binns et al., 2018); (Saxena et al., 2019)	Task type (mechanical vs. human-like) × Algorithmic bias → Perceived fairness	Task type (mechanical vs. human-like) × Perceived fairness → Behavioural responses Task type (high-impact vs. low-impact) × Algorithmic bias → Perceived Fairness Task type (high-impact vs. low-impact) × Perceived fairness → Behavioural responses Type of data subjects × Algorithmic bias → Perceived fairness Type of data subjects × Perceived fairness → Behavioural responses
P5	Springer and Whittaker (2019); Lee, Jain et al. (2019); Binns et al. (2018); Dodge et al. (2019); Ebrahimi and Hassanein (2019); Lee, Jain et al. (2019); Rossi (2019); (Rader et al., 2018)	Algorithmic transparency × Algorithmic bias → Perceived fairness Outcome control × Algorithmic bias → Perceived fairness	Algorithmic transparency × Perceived fairness → Behavioural responses Outcome control × Perceived fairness → Behavioural responses Auditability × Algorithmic bias → Perceived fairness Auditability × Perceived fairness → Behavioural responses
P6	Lee et al. (2019); (Koene et al., 2018); (Koene, 2017; Loi et al., 2019); (Rantavuo, 2019); (Wong, 2019)	N/A	Organisational policies, norms, rules, standards × Algorithmic bias → Perceived fairness Organisational policies, norms, rules, standards × Perceived fairness → Behavioural responses
P7	Almada (2019); Barocas and Selbst (2016); Garcia (2016); (Almada, 2019; Barocas & Selbst, 2016; Garcia, 2016); Kroll et al. (2016)	N/A	Laws and regulations, social norms, culture × Algorithmic bias → Perceived fairness Laws and regulations, social norms, culture × Perceived fairness → Behavioural responses

Figure 2. Theoretical model.

Verma, S., & Rubin, J. (2018). Fairness definitions explained. 2018 IEEE/ACM international workshop on software fairness (FairWare), Gothenburg, Sweden.

 Google Scholar

Shrestha, Y. R., & Yang, Y. (2019). Fairness in algorithmic decision-making: Applications in multi-winner voting, machine learning, and recommender systems. Algorithms, 12(9), 199. https://doi.org/10.3390/a12090199

 Web of Science ®Google Scholar

Lee, M., Kim, J., & Lizarondo, L. (2017). A human-centered approach to algorithmic services: Considerations for fair and motivating smart community service management that allocates donations to non-profit organizations. Proceedings of the 2017 CHI conference on human factors in computing systems, Denver, CO, USA.

 Google Scholar

Saxena, N.-A., Huang, K., DeFilippis, E., Radanovic, G., Parkes, D. C., & Liu, Y. (2019). How do fairness definitions fare? Examining public attitudes towards algorithmic definitions of fairness. Proceedings of the 2019 AAAI/ACM conference on AI, ethics, and society, Honolulu, HI, USA.

 Google Scholar

Ebrahimi, S., & Hassanein, K. (2019). Empowering users to detect data analytics discriminatory recommendations. Proceedings of the 40th International Conference on Information Systems, Munich, Germany.

 Google Scholar

Rossi, F. (2019). Building trust in artificial intelligence. Journal of International Affairs, 72(1), 127–134. https://www.jstor.org/stable/26588348

 Google Scholar

Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K., & Dugan, C. (2019). Explaining models: An empirical study of how explanations impact fairness judgment. Proceedings of the 24th international conference on intelligent user interfaces. Marina del Ray, CA, USA.

 Google Scholar

Lee, M. (2018a). Understanding perception of algorithmic decisions: Fairness, trust, and emotion in response to algorithmic management. Big Data & Society, 5(1), 2053951718756684. https://doi.org/10.1177/2053951718756684

 Web of Science ®Google Scholar

Barlas, P., Kleanthous, S., Kyriakou, K., & Otterbacher, J. (2019). What makes an image tagger fair? Proceedings of the 27th ACM conference on user modeling, adaptation and personalization, Larnaca, Cyprus.

 Google Scholar

Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018). ‘It’s reducing a human being to a percentage’ perceptions of justice in algorithmic decisions. Proceedings of the 2018 CHI conference on human factors in computing systems, Montreal, QC, Canada.

 Google Scholar

Springer, A., & Whittaker, S. (2019). Making transparency clear: The dual importance of explainability and auditability. IUI Workshops.

 Google Scholar

Lee, M., Jain, A., Cha, H., Ojha, S., & Kusbit, D. (2019). Procedural justice in algorithmic fairness: Leveraging transparency and outcome control for fair algorithmic mediation. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1–26.

 PubMedGoogle Scholar

Rader, E., Cotter, K., & Cho, J. (2018). Explanations as mechanisms for supporting algorithmic transparency. Proceedings of the 2018 CHI conference on human factors in computing systems,Montreal, QC, Canada.

 Google Scholar

Lee, M., Kusbit, D., Kahng, A., Kim, J., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., & Psomas, A. (2019). WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), Artcile 181, 1–35. https://doi.org/10.1145/3359283

 PubMedGoogle Scholar

Koene, A., Dowthwaite, L., & Seth, S. (2018). IEEE P7003™ standard for algorithmic bias considerations: Work in progress paper. Proceedings of the international workshop on software fairness, Gothenburg, Sweden.

 Google Scholar

Koene, A. (2017). Algorithmic bias: Addressing growing concerns [leading edge]. IEEE Technology and Society Magazine, 36(2), 31–32. https://doi.org/10.1109/MTS.2017.2697080

 Web of Science ®Google Scholar

Loi, M., Heitz, C., Ferrario, A., Schmid, A., & Christen, M. (2019). Towards an ethical code for data-based business. 2019 6th Swiss Conference on Data Science (SDS), Bern, Switzerland.

 Google Scholar

Rantavuo, H. (2019). Designing for intelligence: User-centred design in the age of algorithms. Proceedings of the 5th International ACM in-cooperation HCI and UX conference,Jakarta Surabaya, Bali, Indonesia.

 Google Scholar

Wong, P.-H. (2019). Democratizing Algorithmic Fairness. Philosophy & Technology 33, 225–244. http://doi.org/10.1007/s13347-019-00355-w

 Google Scholar

Almada, M. (2019). Human intervention in automated decision-making: Toward the construction of contestable systems. Proceedings of the seventeenth international conference on artificial intelligence and law, Montreal, QC, Canada.

 Google Scholar

Barocas, S., & Selbst, A. D. (2016). Big Data’s disparate impact. Social Science Research Network, 62. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2477899#

 Google Scholar

Garcia, M. (2016). Racist in the machine: The disturbing implications of algorithmic bias. World Policy Journal, 33(4), 111–117. https://doi.org/10.1215/07402775-3813015

 Google Scholar

Kroll, J. A., Barocas, S., Felten, E. W., Reidenberg, J. R., Robinson, D. G., & Yu, H. (2016). Accountable algorithms. U. Pa. L. Review, 165(3), 633. https://scholarship.law.upenn.edu/penn_law_review/vol165/iss3/3/

 Web of Science ®Google Scholar