Development and Validation of an Algorithm Literacy Scale for Internet Users

References

• Ananny, M., & Crawford, K. (2018). Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society, 20(3), 973–989. https://doi.org/https://doi.org/10.1177/1461444816676645
   Web of Science ®Google Scholar
• Ball, K., & Webster, W. (2020). Big data and surveillance: Hype, commercial logics and new intimate spheres. Big Data & Society, 7(1), 205395172092585. https://doi.org/https://doi.org/10.1177/2053951720925853
   Web of Science ®Google Scholar
• Beatty, P., Herrmann, D., Puskar, C., & Kerwin, J. (1998). “Don’t know” responses in surveys: Is what I know what you want to know and do I want you to know it? Memory (Hove, England), 6(4), 407–426. https://doi.org/https://doi.org/10.1080/741942605
   PubMed Web of Science ®Google Scholar
• Bishop, S. (2019). Managing visibility on YouTube through algorithmic gossip. New Media & Society, 21(11-12), 2589–2606. https://doi.org/https://doi.org/10.1177/1461444819854731
   Google Scholar
• Bond, T. G., & Fox, C. F. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Routledge.
   Google Scholar
• Brodsky, J. E., Zomberg, D., Powers, K. L., & Brooks, P. J. (2020). Assessing and fostering college students’ algorithm awareness across online contexts. Journal of Media Literacy Education, 12(3), 43–57. https://doi.org/https://doi.org/10.23860/JMLE-2020-12-3-5
   Google Scholar
• Bucher, T. (2012). Want to be on the top? Algorithmic power and the threat of invisibility on Facebook. New Media & Society, 14(7), 1164–1180. https://doi.org/https://doi.org/10.1177/1461444812440159
   Web of Science ®Google Scholar
• Bucher, T. (2017). The algorithmic imaginary: Exploring the ordinary affects of Facebook algorithms. Information, Communication & Society, 20(1), 30–44. https://doi.org/https://doi.org/10.1080/1369118X.2016.1154086
   Web of Science ®Google Scholar
• Burrell, J., Kahn, Z., Jonas, A., & Griffin, D. (2019). When users control the algorithms. In A. Lampinen, D. Gergle, & D. A. Shamma (Eds.), Proceedings of the ACM on human-computer interaction (Vol. 3, pp. 1–20). Association for Computing Machinery Inc. (ACM). https://doi.org/https://doi.org/10.1145/3359240
   Google Scholar
• Carpenter, S. (2018). Ten steps in scale development and reporting: A guide for researchers. Communication Methods and Measures, 12(1), 25–44. https://doi.org/https://doi.org/10.1080/19312458.2017.1396583
   Web of Science ®Google Scholar
• Chen, D.‑T.,Wu, J., & Wang, Y. (2011). Unpacking new media literacy. Journal of Systemics, Cybernetics and Informatics, 9(2), 84–88. http://www.iiisci.org/journal/sci/FullText.asp?var=&id=OL508KR
   Google Scholar
• Claes, A., & Philippette, T. (2020). Defining a critical data literacy for recommender systems: A media-grounded approach. Journal of Media Literacy Education, 12(3), 17–29. https://doi.org/https://doi.org/10.23860/JMLE-2020-12-3-3
   Google Scholar
• Coombs, C. H., Coombs, L. C. (1976). “Don’t know”: Item ambiguity or respondent uncertainty? Public Opinion Quarterly, 40(4), 497. https://doi.org/https://doi.org/10.1086/268336
   Web of Science ®Google Scholar
• Cotter, K. (2019). Playing the visibility game: How digital influencers and algorithms negotiate influence on Instagram. New Media & Society, 21(4), 895–913. https://doi.org/https://doi.org/10.1177/1461444818815684
   Web of Science ®Google Scholar
• Cotter, K., & Reisdorf, B. C. (2020). Algorithmic knowledge gaps: A new dimension of (Digital) inequality. International Journal of Communication (Ijoc), 2020(14), 745–765. https://ijoc.org/index.php/ijoc/article/view/12450
   Google Scholar
• Cotter, K. M. (2020). Critical algorithmic literacy: Power, epistemology, and platforms [Dissertation]. Michigan State University, Michigan. https://search.proquest.com/openview/3d5766d511ea8a1ffe54c53011acf4f2/1?pq-origsite=gscholar&cbl=18750&diss=y
   Google Scholar
• DeMars, C. (2010). Item response theory. Oxford University Press.
   Google Scholar
• Dogruel, L., Facciorusso, D., & Stark, B. (2020). ‘I’m still the master of the machine.’ Internet users’ awareness of algorithmic decision-making and their perception of its effect on their autonomy. Special Issue of Digital Communication Research.Information, Communication & Society. Advance online publication https://doi.org/https://doi.org/10.1080/1369118X.2020.1863999
   Google Scholar
• Dogruel, L. (2021). What is algorithm literacy? A conceptualization and challenges of its empirical measurement. In C. Schumann & M. Taddicken (Eds.), Algorithms and Communication. Special Issue of Digital Communication Research.
   Google Scholar
• Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum.
   Google Scholar
• Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015). I always assumed that I wasn’t really that close to [her]. In B. Begole, J. Kim, K. Inkpen, & W. Woo ( Chairs), The 33rd annual ACM conference, Seoul, Republic of Korea.
   Google Scholar
• Fletcher, R., & Nielsen, R. K. (2018). Automated serendipity. Digital Journalism, 6(8), 976–989. https://doi.org/https://doi.org/10.1080/21670811.2018.1502045
   Web of Science ®Google Scholar
• Gillespie, T. (2014). Relevance of algorithms. In T. Gillespie, P. J. Boczkowski, & K. A. Foot (Eds.), Inside technology. Media technologies: Essays on communication, materiality, and society (pp. 167–194). The MIT Press.
   Google Scholar
• Gran, A. ‑. B., Booth, P., & Bucher, T. (2021). To be or not to be algorithm aware: A question of a new digital divide? Information, Communication & Society, 24(12), 1779–1796. https://doi.org/https://doi.org/10.1080/1369118X.2020.1736124
   Google Scholar
• Hallinan, B., & Striphas, T. (2016). Recommended for you: The Netflix prize and the production of algorithmic culture. New Media & Society, 18(1), 117–137. https://doi.org/https://doi.org/10.1177/1461444814538646
   Web of Science ®Google Scholar
• Hamilton, K., Karahalios, K., Sandvig, C., & Eslami, M. (2014). A path to understanding the effects of algorithm awareness. CHI’14 Extended Abstracts on Human Factors in Computing Systems, 631–642. https://doi.org/https://doi.org/10.1145/2559206.2578883
   Google Scholar
• Hancock, J. T., Naaman, M., & Levy, K. (2020). AI-mediated communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication, 25(1), 89–100. https://doi.org/https://doi.org/10.1093/jcmc/zmz022
   Web of Science ®Google Scholar
• Hargittai, E., Gruber, J., Djukaric, T., Fuchs, J., & Brombach, L. (2020). Black box measures? How to study people’s algorithm skills. Information, Communication & Society, 23(5), 764–775. https://doi.org/https://doi.org/10.1080/1369118X.2020.1713846
   Google Scholar
• Helberger, N. (2019). On the democratic role of news recommenders. Digital Journalism, 7(8), 993–1012. https://doi.org/https://doi.org/10.1080/21670811.2019.1623700
   Web of Science ®Google Scholar
• Hobbs, R. (2020). Propaganda in an age of algorithmic personalization: Expanding literacy research and practice. Reading Research Quarterly, 55(3), 521–533. https://doi.org/https://doi.org/10.1002/rrq.301
   Web of Science ®Google Scholar
• Katzenbach, C., & Ulbricht, L. (2019). Algorithmic governance. Internet Policy Review, 8(4). https://doi.org/https://doi.org/10.14763/2019.4.1424
   Web of Science ®Google Scholar
• Kemper, J., & Kolkman, D. (2018). Transparent to whom? No algorithmic accountability without a critical audience. Information, Communication & Society, 19(4), 1–16. https://doi.org/https://doi.org/10.1080/1369118X.2018.1477967
   Google Scholar
• Kitchin, R. (2017). Thinking critically about and researching algorithms. Information, Communication & Society, 20(1), 14–29. https://doi.org/https://doi.org/10.1080/1369118X.2016.1154087
   Web of Science ®Google Scholar
• Klawitter, E., & Hargittai, E. (2018). “It’s like learning a whole other language:” The role of algorithmic skills in the curation of creative goods. International Journal of Communication, 12(2018), 3490–3510. https://ijoc.org/index.php/ijoc/article/view/7864
   Google Scholar
• Koltay, T. (2011). The media and the literacies: Media literacy, information literacy, digital literacy. Media, Culture & Society, 33(2), 211–221. https://doi.org/https://doi.org/10.1177/0163443710393382
   Web of Science ®Google Scholar
• Latzer, M., Hollnbuchner, K., Just, N., & Saurwein, F. (2016). The economics of algorithmic selection on the internet. In J. M. Bauer & M. Latzer (Eds.), Handbook on the economics of the internet (pp. 395–424). Edward Elgar Publishing.
   Google Scholar
• Latzer, M., & Festic, N. (2019). A guideline for understanding and measuring algorithmic governance in everyday life. Internet Policy Review, 8(2). https://doi.org/https://doi.org/10.14763/2019.2.1415
   Web of Science ®Google Scholar
• Latzer, M., Festic, N., & Kappeler, K. (2020). Awareness of algorithmic selection and attitudes in Switzerland. Report 2 from the project: The significance of algorithmic selection for everyday life: The case of Switzerland. University of Zurich. http://mediachange.ch/research/algosig
   Google Scholar
• Leong, L. (2020). Domesticating algorithms: An exploratory study of Facebook users in Myanmar. The Information Society, 36(2), 97–108. https://doi.org/https://doi.org/10.1080/01972243.2019.1709930
   Web of Science ®Google Scholar
• Linacre, J. M. (2002). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions, 16(2), 878. https://www.rasch.org/rmt/rmt162f.htm
   Google Scholar
• Livingstone, S., Stoilova, M., & Nandagiri, R. (2020). Data and privacy literacy. The role of the school in educating children in a datafied society. In D. Frau‐Meigs, S. Kotilainen, M. Pathak‐Shelat, M. Hoechsmann, & S. R. Poyntz (Eds.), The handbook of media education research (pp. 413–426). Wiley.
   Google Scholar
• Lutz, C. (2019). Digital inequalities in the age of artificial intelligence and big data. Human Behavior and Emerging Technologies, 1(2), 141–148. https://doi.org/https://doi.org/10.1002/hbe2.140
   Google Scholar
• Masur, P. K. (2018). Situational privacy and self-disclosure: Communication processes in online environments. Cham, Switzerland: Springer.
   Google Scholar
• Masur, P. K. (2020). How online privacy literacy supports self-data protection and self-determination in the age of information. Media and Communication, 8(2), 258–269. https://doi.org/https://doi.org/10.17645/mac.v8i2.2855
   Web of Science ®Google Scholar
• Pangrazio, L., & Selwyn, N. (2019). ‘Personal data literacies’: A critical literacies approach to enhancing understandings of personal digital data. New Media & Society, 21(2), 419–437. https://doi.org/https://doi.org/10.1177/1461444818799523
   Web of Science ®Google Scholar
• Potter, W. J. (2010). The state of media literacy. Journal of Broadcasting & Electronic Media, 54(4), 675–696. https://doi.org/https://doi.org/10.1080/08838151.2011.521462
   Web of Science ®Google Scholar
• Potter, W. J., & Thai, C. L. (2019). Reviewing media literacy intervention studies for validity. Review of Communication Research, 7, 38–66. https://doi.org/https://doi.org/10.12840/ISSN.2255-4165.018
   Google Scholar
• Powers, E. (2017). My news feed is filtered? Digital Journalism, 5(10), 1315–1335. https://doi.org/https://doi.org/10.1080/21670811.2017.1286943
   Web of Science ®Google Scholar
• Prey, R. (2017). Nothing personal: Algorithmic individuation on music streaming platforms. Media, Culture & Society, 10(1), 016344371774514. https://doi.org/https://doi.org/10.1177/0163443717745147
   Google Scholar
• Rader, E., & Gray, R. (2015). Understanding user beliefs about algorithmic curation in the Facebook news feed. In Begole, J. Kim, K. Inkpen, & W. Woo (Eds.), Proceedings of the 33rd annual ACM conference on human factors in computing systems (pp. 173–182). ACM: New York.  https://doi.org/https://doi.org/10.1145/2702123.2702174
   Google Scholar
• Rozendaal, E., Lapierre, M. A., van Reijmersdal, E. A., & Buijzen, M. (2011). Reconsidering advertising literacy as a defense against advertising effects. Media Psychology, 14(4), 333–354. https://doi.org/https://doi.org/10.1080/15213269.2011
   Google Scholar
• Schwartz, S. A., & Mahnke, M. S. (2020). Facebook use as a communicative relation: Exploring the relation between Facebook users and the algorithmic news feed. Information, Communication & Society. https://doi.org/https://doi.org/10.1080/1369118X.2020.1718179
   Google Scholar
• Sharabi, L. L. (2020). Exploring how beliefs about algorithms shape (Offline) success in online dating: A two-wave longitudinal investigation. Communication Research, 009365021989693. https://doi.org/https://doi.org/10.1177/0093650219896936
   Google Scholar
• Shin, D., & Park, Y. J. (2019). Role of fairness, accountability, and transparency in algorithmic affordance. Computers in Human Behavior, 98(September), 277–284. https://doi.org/https://doi.org/10.1016/j.chb.2019.04.019
   Web of Science ®Google Scholar
• Sundar, S. S. (2020). Rise of machine agency: A framework for studying the psychology of human–AI interaction (HAII). Journal of Computer-Mediated Communication, 25(1), 74–88. https://doi.org/https://doi.org/10.1093/jcmc/zmz026
   Web of Science ®Google Scholar
• Susser, D., Roessler, B., & Nissenbaum, H. (2019). Technology, autonomy, and manipulation. Internet Policy Review, 8(2). https://doi.org/https://doi.org/10.14763/2019.2.1410
   Web of Science ®Google Scholar
• Thorson, K. (2020). Attracting the news: Algorithms, platforms, and reframing incidental exposure. Journalism: Theory, Practice & Criticism, 1067–1082. https://doi.org/https://doi.org/10.1177/1464884920915352
   Google Scholar
• Trepte, S., Teutsch, D., Masur, P. K., Eicher, C., Fischer, M., Hennhöfer, A., & Lind, F. (2015). Do people know about privacy and data protection strategies? Towards the “online privacy literacy scale” (OPLIS). In S. Gutwirth, R. Leenes, & P. de Hert (Eds.), Law, governance and technology series: Vol. 20. Reforming European data protection law (pp. 333–366). Springer.
   Google Scholar
• Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. MESA Press.
   Google Scholar
• Wright, B. D., & Stone, M. H. (1999). Measurement Essentials. Wide Range, Inc.
   Google Scholar
• Yeung, K. (2018). Algorithmic regulation: A critical interrogation. Regulation & Governance, 12(4), 505–523. https://doi.org/https://doi.org/10.1111/rego.12158
   Web of Science ®Google Scholar
• Zarouali, B., Boerman, S. C., & de Vreese, C. H. (2021). Is this recommended by an algorithm? The development and validation of the algorithmic media content awareness scale (AMCA-scale). Telematics and Informatics, 62. https://doi.org/https://doi.org/10.1016/j.tele.2021.101607
   Google Scholar