Topicalizer: reframing core concepts in machine learning visualization by co-designing for interpretivist scholarship

References

• Ackerman, M. S. (2000, September). The intellectual challenge of CSCW: The gap between social requirements and technical feasibility. Human–Computer Interaction, 15(2–3), 179–203. https://doi.org/10.1207/S15327051HCI15235
   Web of Science ®Google Scholar
• Alexander, E., Kohlmann, J., Valenza, R., Witmore, M., & Gleicher, M. (2014, October). Serendip: Topic model-driven visual exploration of text corpora. In Conference on Visual Analytics Science and Technology (VAST) (p. 173–182). IEEE. https://doi.org/10.1109/VAST.2014.7042493
   Google Scholar
• Ananny, M. (2011, April). The curious connection between apps for gay men and sex offenders. The Atlantic.
   Google Scholar
• Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016, May). Machine bias. Pro Publica.
   Google Scholar
• Bardzell, S. (2010). Feminist HCI: Taking stock and outlining an agenda for design. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (pp. 1301–1310) ACM.
   Google Scholar
• Barocas, S., & Selbst, A. D. (2016). Big data’s disparate impact. California Law Review, 104(3), 671–732. http://dx.doi.org/10.15779/Z38BG31
   Web of Science ®Google Scholar
• Baumer, E. P. S. (2017). Toward human-centered algorithm design. Big Data & Society, 4(2). https://doi.org/10.1177/2053951717718854
   PubMed Web of Science ®Google Scholar
• Baumer, E. P. S., Guha, S., Quan, E., Mimno, D., & Gay, G. K. (2015). Missing photos, suffering withdrawal, or finding freedom? How experiences of social media non-use influence the likelihood of reversion. Social Media + Society, 1(2). https://doi.org/10.1177/2056305115614851
   Web of Science ®Google Scholar
• Baumer, E. P. S., & McGee, M. (2019). Speaking on behalf of: Representation, delegation, and authority in computational text analysis. In Proceedings of the AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES) (p. 8). AAAI/ACM.
   Google Scholar
• Baumer, E. P. S., Mimno, D., Guha, S., Quan, E., & Gay, G. K. (2017, June). Comparing grounded theory and topic modeling: Extreme divergence or unlikely convergence? Journal of the Association for Information Science and Technology (JASIST), 68(6), 1397–1410. https://doi.org/10.1002/asi.23786
   Web of Science ®Google Scholar
• Berger, P. L., & Luckmann, T. (1966). The social construction of reality: A treatise in the sociology of knowledge. Anchor Books.
   Google Scholar
• Bhatia, S., Lau, J. H., & Baldwin, T. (2016, December). Automatic labelling of topics with neural embeddings. In Proceedings of the International Conference on Computational Linguistics (COLING) (pp. 953–963). https://doi.org/10.1016/j.arth.2016.09.010
   Google Scholar
• Blei, D. M. (2012, April). Probabilistic topic models. Communications of the ACM, 55(4), 77–84. https://doi.org/10.1145/2133806.2133826
   Web of Science ®Google Scholar
• Blei, D. M., & Lafferty, J. D. (2007a). A correlated topic model of science. The Annals of Applied Statistics, 1(1), 17–35. https://doi.org/10.1214/07-AOAS114
   Web of Science ®Google Scholar
• Blei, D. M., & Lafferty, J. D. (2007b). Modeling science. http://www.cs.cmu.edu/˜lemur/science/
   Google Scholar
• Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3(Jan), 993–1022. http://www.jmlr.org/papers/v3/blei03a.html
   Google Scholar
• Boehner, K., Vertesi, J., Sengers, P., & Dourish, P. (2007). How HCI interprets the probes. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (pp. 1077–1086). ACM. https://doi.org/10.1145/1240624.1240789
   Google Scholar
• Boyd-Graber, J., Hu, Y., & Mimno, D. (2017, July). Applications of topic models. Foundations and Trends ® in Information Retrieval, 11(2–3), 143–296. https://doi.org/10.1561/1500000030
   Web of Science ®Google Scholar
• Boydstun, A. E., Card, D., Gross, J. H., Resnik, P., & Smith, N. A. (2014). Tracking the development of media frames within and across policy issues. In Annual Meeting of the American Political Science Association (APSA).
   Google Scholar
• Bucher, T. (2017). The algorithmic imaginary: Exploring the ordinary affects of Facebook algorithms. Information, Communication & Society, 20(1), 30–44. https://doi.org/10.1080/1369118X.2016.1154086
   Web of Science ®Google Scholar
• Burrell, J. (2016, January). How the machine ‘thinks’: Understanding opacity in machine learning algorithms. Big Data & Society, 3(1), 2053951715622512. https://doi.org/10.1177/2053951715622512
   Web of Science ®Google Scholar
• Chaney, A. J. B. (2012). Wikipedia topics. http://www.princeton.edu/˜achaney/tmve/wiki100k/browse/topic-presence.html
   Google Scholar
• Chaney, A. J. B., & Blei, D. M. (2012). Visualizing topic models. In Proceedings of the International AAAI Conference on Weblogs and Social Media (ICWSM) (p. 419–422). AAAI.
   Google Scholar
• Chang, J., Gerrish, S., Wang, C., Boyd-Graber, J. L., & Blei, D. M. (2009). Reading tea leaves: How humans interpret topic models. In Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, & A. Culotta (Eds.), Advances in Neural Information Processing Systems (NIPS) (pp. 288–296). Curran Associates.
   Google Scholar
• Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. SAGE Publications.
   Google Scholar
• Chen, N.-C., Drouhard, M., Kocielnik, R., Suh, J., & Aragon, C. R. (2018, June). Using machine learning to support qualitative coding in social science: Shifting the focus to ambiguity. ACM Transactions on Interactive Intelligent Systems, 8(2), 9:1–9: 20. https://doi.org/10.1145/3185515
   PubMed Web of Science ®Google Scholar
• Choo, J., Lee, C., Reddy, C. K., & Park, H. (2013, December). UTOPIAN: User-driven topic modeling based on interactive nonnegative matrix factorization. IEEE Transactions on Visualization and Computer Graphics, 19(12), 1992–2001. https://doi.org/10.1109/TVCG.2013.212
   PubMed Web of Science ®Google Scholar
• Christin, A. (2017, December). Algorithms in practice: Comparing web journalism and criminal justice. Big Data & Society, 4(2), 2053951717718855. https://doi.org/10.1177/2053951717718855
   Web of Science ®Google Scholar
• Chuang, J., Manning, C. D., & Heer, J. (2012). Termite: Visualization techniques for assessing textual topic models. In Proceedings of the International Working Conference on Advanced Visual Interfaces (AVI) (pp. 74–77). ACM. https://doi.org/10.1145/2254556.2254572
   Google Scholar
• Chuang, J., Ramage, D., Manning, C., & Heer, J. (2012). Interpretation and trust: Designing model-driven visualizations for text analysis. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (pp. 443–452). ACM. https://doi.org/10.1037/a0030967
   Google Scholar
• Chuang, J., Roberts, M. E., Stewart, B. M., Weiss, R., Tingley, D., Grimmer, J., & Heer, J. (2015). TopicCheck: Interactive alignment for assessing topic model stability. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) (pp. 175–184).
   Google Scholar
• Clement, T., Tcheng, D., Auvil, L., Capitanu, B., & Barbosa, J. (2013, December). Distant listening to gertrude Stein’s ‘Melanctha’: Using similarity analysis in a discovery paradigm to analyze prosody and author influence. Literary and Linguistic Computing, 28(4), 582–602. https://doi.org/10.1093/llc/fqt040
   Google Scholar
• Concannon, S. J., Balaam, M., Simpson, E., & Comber, R. (2018). Applying computational analysis to textual data from the wild: A feminist perspective. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (pp. 226: 1–226:13). ACM. https://doi.org/10.1145/3173574.3173800
   Google Scholar
• Corbin, J., & Strauss, A. (n.d.). Grounded theory research: Procedures, canons, and evaluative criteria. Qualitative Sociology, 13(1), 3. https://doi.org/10.1007/BF00988593
   Google Scholar
• D’Ignazio, C., & Klein, L. F. (2016). Feminist data visualization. In Workshop on Visualization for the Digital Humanities, at IEEE VIS Conference.
   Google Scholar
• DeVito, M. A., Birnholtz, J., Hancock, J. T., French, M., & Liu, S. (2018). How people form folk theories of social media feeds and what it means for how we study self-presentation. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (pp. 120: 1–120:12). ACM. https://doi.org/10.1145/3173574.3173694
   Google Scholar
• DeVito, M. A., Hancock, J. T., French, M., Birnholtz, J., Antin, J., Karahalios, K., Tong, S., & Shklovski, I. (2018). The algorithm and the user: How can HCI use lay understandings of algorithmic systems? In Extended Abstracts of the ACM Conference on Human Factors in Computing Systems (CHI EA) (pp. panel04:1–panel04:6). ACM. https://doi.org/10.1145/3170427.3186320
   Google Scholar
• Dinakar, K., Chen, J., Lieberman, H., Picard, R., & Filbin, R. (2015). Mixed-initiative real-time topic modeling & visualization for crisis counseling. In Proceedings of the ACM Conference on Intelligent User Interfaces (IUI) (pp. 417–426). ACM. https://doi.org/10.1145/2678025.2701395
   Google Scholar
• Došilović, F. K., Brčić, M., & Hlupić, N. (2018, May). Explainable artificial intelligence: A survey. In 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) (pp. 0210–0215). https://doi.org/10.23919/MIPRO.2018.8400040
   Google Scholar
• Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017). UX design innovation: Challenges for working with machine learning as a design material. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (pp. 278–288). ACM. https://doi.org/10.1145/3025453.3025739
   Google Scholar
• Drouhard, M., Chen, N. C., Suh, J., Kocielnik, R., Pen˜ a-Araya, V., Cen, K., Zheng, X., & Aragon, C. R. (2017, April). Aeonium: Visual analytics to support collaborative qualitative coding. In IEEE Pacific Visualization Symposium (PacificVis) (p. 220–229). IEEE. https://doi.org/10.1109/PACIFICVIS.2017.8031598
   Google Scholar
• Ehn, P. (1988). Work-oriented design of computer artifacts. Lawrence Erlbaum Associates.
   Google Scholar
• Eisenstein, J., Chau, D. H., Kittur, A., & Xing, E. (2012). TopicViz: Interactive topic exploration in document collections. In Extended Abstracts of the ACM Conference on Human Factors in Computing Systems (CHI EA) (pp. 2177–2182). ACM. https://doi.org/10.1145/2212776.2223772
   Google Scholar
• Eslami, M., Karahalios, K., Sandvig, C., Vaccaro, K., Rickman, A., Hamilton, K., & Kirlik, A. (2016). First I ”Like” It, then I Hide It: Folk theories of social feeds. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (pp. 2371–2382). ACM. https://doi.org/10.1145/2858036.2858494
   Google Scholar
• Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015). ”I always assumed that I wasn’t really that close to [Her]”: Reasoning about invisible algorithms in news feeds. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (pp. 153–162). ACM. https://doi.org/10.1145/2702123.2702556
   Google Scholar
• Foucault, M. (1972). The archaeology of knowledge. ( A. M. Sheridan Smith, Trans). Pantheon Books. https://doi.org/10.1002/9780470776407.ch20.
   Google Scholar
• Frederickson, B. (2019, June). Area proportional Venn and Euler diagrams in JavaScript: Benfred/venn.js.
   Google Scholar
• Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.
   Google Scholar
• Gillespie, T. (2012, February). Can an algorithm be wrong? Limn, 1(2). https://limn.it/articles/can-an-algorithm-be-wrong/
   Google Scholar
• Gillespie, T. (2013). The relevance of algorithms. In T. Gillespie, P. Bockzkowski, & K. Foot (Eds.), Media technologies. pp. 167-193, MIT Press.
   Google Scholar
• Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory. Routledge.
   Google Scholar
• Goldstone, A. (n.d.). Topic model browser. http://agoldst.github.io/dfr-browser/demo/
   Google Scholar
• Goldstone, A., & Underwood, T. (2014, November). The quiet transformations of literary studies: What thirteen thousand scholars could tell us. New Literary History, 45(3), 359–384. https://doi.org/10.1353/nlh.2014.0025
   Web of Science ®Google Scholar
• Greenberg, S., & Buxton, B. (2008). Usability evaluation considered harmful (Some of the Time). In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (pp. 111–120). ACM. https://doi.org/10.1111/j.1399-6576.2007.01487.x
   Google Scholar
• Isenberg, T., Isenberg, P., Chen, J., Sedlmair, M., & Möller, T. (2013, December). A systematic review on the practice of evaluating visualization. IEEE Transactions on Visualization and Computer Graphics, 19(12), 2818–2827. https://doi.org/1109/TVCG.2013.126
   PubMed Web of Science ®Google Scholar
• Jockers, M. L. (2013). Macroanalysis: Digital methods & literary history. University of Illinois Press.
   Google Scholar
• Jockers, M. L. (n.d.). Expanded stopwords list. http://www.matthewjockers.net/macroanalysisbook/expanded-stopwords-list/.
   Google Scholar
• Jockers, M. L., & Mimno, D. (2013, December). Significant themes in 19th-century literature. Poetics, 41(6), 750–769. https://doi.org/10.1016/j.poetic.2013.08.005
   Web of Science ®Google Scholar
• Klein, L. F., & Eisenstein, J. (2013, December). Reading Thomas Jefferson with TopicViz: Towards a thematic method for exploring large cultural archives. Scholarly and Research Communication, 4(3), https://doi.org/10.22230/src.2013v4n3a121
   Google Scholar
• Klein, L. F., Eisenstein, J., & Sun, I. (2015, December). Exploratory thematic analysis for digitized archival collections. Digital Scholarship in the Humanities, 30(suppl 1), i130–i141. https://doi.org/10.1093/llc/fqv052
   Google Scholar
• Knigge, L., & Cope, M. (2006, November). Grounded visualization: Integrating the analysis of qualitative and quantitative data through grounded theory and visualization. Environment and Planning A: Economy and Space, 38(11), 2021–2037. https://doi.org/10.1068/a37327
   Web of Science ®Google Scholar
• Kok, P., Baiker, M., Hendriks, E. A., Post, F. H., Dijkstra, J., Lowik, C. W. G. M., Lelieveldt, B. P. F., & Botha, C. P. (2010, November). Articulated planar reformation for change visualization in small animal imaging. IEEE Transactions on Visualization and Computer Graphics, 16(6), 1396–1404. https://doi.org/10.1109/TVCG.2010.134
   PubMed Web of Science ®Google Scholar
• Kuhn, S., & Muller, M. J. (1993, June). Participatory design. Communications of the ACM, 36(4), 24–28. https://doi.org/10.1145/153571.255960
   Google Scholar
• Lam, H., Bertini, E., Isenberg, P., Plaisant, C., & Carpendale, S. (2012, September). Empirical studies in information visualization: Seven scenarios. IEEE Transactions on Visualization and Computer Graphics, 18(9), 1520–1536. https://doi.org/10.1109/TVCG.2011.279
   PubMed Web of Science ®Google Scholar
• Lau, J. H., & Baldwin, T. (2016, June). The sensitivity of topic coherence evaluation to topic cardinality. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) (pp. 483–487). Association for Computational Linguistics.
   Google Scholar
• Lau, J. H., Grieser, K., Newman, D., & Baldwin, T. (2011). Automatic labelling of topic models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) (pp. 1536–1545). Association for Computational Linguistics.
   Google Scholar
• Lau, J. H., Newman, D., & Baldwin, T. (2014, April). Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the Annual Meeting of the European Chapter of the Association for Computational Linguistics (EACL) (pp. 530–539). Association for Computational Linguistics.
   Google Scholar
• Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., & Jebara, T. (2009, February). Computational social science. Science, 323(5915), 721–723. https://doi.org/10.1126/science.1167742
   PubMed Web of Science ®Google Scholar
• Leahu, L. (2016). Ontological surprises: A relational perspective on machine learning. In Proceedings of the ACM Conference on Designing Interactive Systems (DIS) (pp. 182–186). ACM. https://doi.org/10.1145/2901790.2901840
   Google Scholar
• Leahu, L., Schwenk, S., & Sengers, P. (2008). Subjective objectivity: Negotiating emotional meaning. In Proceedings of the ACM Conference on Designing Interactive Systems (DIS) (pp. 425–434). ACM. https://doi.org/10.1145/1394445.1394491
   Google Scholar
• Leahu, L., & Sengers, P. (2014). Freaky: Performing hybrid human-machine emotion. In Proceedings of the ACM Conference on Designing Interactive Systems (DIS) (pp. 607–616). ACM. https://doi.org/10.1145/2598510.2600879
   Google Scholar
• Leahu, L., Sengers, P., & Mateas, M. (2008). Interactionist AI and the promise of Ubicomp, or, how to put your box in the world without putting the world in your box. In Proceedings of the International Conference on Ubiquitous Computing (Ubicomp) (pp. 134–143). ACM. https://doi.org/10.1145/1409635.1409654
   Google Scholar
• Lee, H., Kihm, J., Choo, J., Stasko, J., & Park, H. (2012, June). iVisClustering: An interactive visual document clustering via topic modeling. Computer Graphics Forum, 31(3pt3), 1155–1164. https://doi.org/10.1111/j.1467-8659.2012.03108.x
   Google Scholar
• Lee, T. Y., Smith, A., Seppi, K., Elmqvist, N., Boyd-Graber, J., & Findlater, L. (2017, September). The human touch: How non-expert users perceive, interpret, and fix topic models. International Journal of Human-computer Studies, 105, 28–42. https://doi.org/10.1016/j.ijhcs.2017.03.007
   Web of Science ®Google Scholar
• Lertvittayakumjorn, P., & Toni, F. (2019, November). Human-grounded evaluations of explanation methods for text classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 5195–5205). ACL. https://doi.org/10.18653/v1/D19-1523
   Google Scholar
• Lind, R. A., & Salo, C. (2002, January). The framing of feminists and feminism in news and public affairs programs in U.S. electronic media. Journal of Communication, 52(1), 211–228. https://doi.org/10.1111/j.1460-2466.2002.tb02540.x
   Web of Science ®Google Scholar
• Lisa Marie Rhody. 2013. Topic Modeling and Figurative Language. Journal of Digital Humanities 2, 1. http://journalofdigitalhumanities.org/2-1/topic-modeling-and-figurative-language-by-lisa-m-rhody/
   Google Scholar
• Lofland, J., Snow, D. A., Anderson, L., & Lofland, L. H. (1971/2005). Analyzing social settings. Wadsworth.
   Google Scholar
• Marathe, M., & Toyama, K. (2018). Semi-automated coding for qualitative research: A user-centered inquiry and initial prototypes. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (pp. 348: 1–348:12). ACM. https://doi.org/10.1145/3173574.3173922
   Google Scholar
• Mei, Q., Shen, X., & Zhai, C. (2007). Automatic labeling of multinomial topic models. In Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD) (pp. 490–499). ACM.
   Google Scholar
• Mimno, D. (2013). jsLDA: In-browser topic modeling. https://mimno.infosci.cornell.edu/jsLDA/index.html
   Google Scholar
• Muller, M. J. (2012). Participatory design: The third space in HCI. In A. Sears & J. A. Jacko (Eds.), The human-computer interaction handbook (Vol. 4235, pp. 1061–1081). Lawrence Erlbaum Associates.
   Google Scholar
• Muller, M. J., Guha, S., Baumer, E. P. S., Mimno, D., & Shami, N. S. (2016). Machine learning and grounded theory method: Convergence, divergence, and combination. In Proceedings of the ACM Conference on Supporting Group Work (GROUP).
   Google Scholar
• Newman, D., Lau, J. H., Grieser, K., & Baldwin, T. (2010). Automatic evaluation of topic coherence. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) (pp. 100–108). ACL.
   Google Scholar
• Orlikowski, W. J., & Baroudi, J. J. (1991). Studying information technology in organizations: Research approaches and assumptions. Information Systems Research, 2(1), 1–28. https://doi.org/10.1287/isre.2.1.1
   Web of Science ®Google Scholar
• Pang, B., & Lee, L. (2008, January). Opinion mining and sentiment analysis. Foundations and Trends in Information, 2(1–2), 1–135. https://doi.org/10.1561/1500000011
   Google Scholar
• Passi, S., & Jackson, S. (2017). Data vision: Learning to see through algorithmic abstraction. In Proceedings of the ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW) (pp. 2436–2447). ACM. https://doi.org/10.1145/2998181.2998331
   Google Scholar
• Perer, A., & Shneiderman, B. (2008). Integrating statistics and visualization: Case studies of gaining clarity during exploratory data analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 265–274). ACM. https://doi.org/10.1145/1357054.1357101
   Google Scholar
• Pinch, T. J., & Bijker, W. E. (1987). The social construction of facts and artifacts. In W. E. Bijker, T. P. Hughes, & T. J. Pinch (Eds.), The social construction of technological systems (pp. 17–50). MIT Press.
   Google Scholar
• Poursabzi-Sangdeh, F., Boyd-Graber, J., Findlater, L., & Seppi, K. (2016). ALTO: Active learning with topic overviews for speeding label induction and document labeling. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) (pp. 1158–1169). Association for Computational Linguistics.
   Google Scholar
• Rhody, L. M. (2017, May). Beyond Darwinian distance: Situating distant reading in a feminist Ut Pictura Poesis Tradition. PMLA, 132(3), 659–667. https://doi.org/10.1632/pmla.2017.132.3.659
   Google Scholar
• Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G. (2014, October). Structural topic models for open-ended survey responses. American Journal of Political Science, 58(4), 1064–1082. https://doi.org/10.1111/ajps.12103
   Web of Science ®Google Scholar
• Röder, M., Both, A., & Hinneburg, A. (2015). Exploring the space of topic coherence measures. In Proceedings of the ACM Conference on Web Search and Data Mining (WSDM) (pp. 399–408). ACM. https://doi.org/10.1145/2684822.2685324
   Google Scholar
• Rosen-Zvi, M., Griffiths, T., Steyvers, M., & Smyth, P. (2004). The author-topic model for authors and documents. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI) (pp. 487–494). AUAI Press.
   Google Scholar
• Samek, W., Wiegand, T., & Müller, K.-R. (2017, August). Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. https://arxiv.org/abs/1708.08296v1
   Google Scholar
• Schofield, A., Magnusson, M., & Mimno, D. (2017). Pulling out the stops: Rethinking stopword removal for topic models. In Proceedings of the Annual Meeting of the European Chapter of the Association for Computational Linguistics (EACL) (Vol. 2, pp. 432–436). ACL.
   Google Scholar
• Schofield, A., & Mimno, D. (2016, July). Comparing apples to apple: The effects of stemmers on topic models. Transactions of the Association for Computational Linguistics, 4, 287–300. https://doi.org/10.1162/tacl_a_00099
   Google Scholar
• Sedlmair, M., Isenberg, P., Isenberg, T., Mahyar, N., & Lam, H. (2016). Beyond time and errors: Novel evaluation methods for visualization (BELIV 2016). In Workshop at IEEE VIS Week.
   Google Scholar
• Sedlmair, M., Meyer, M., & Munzner, T. (2012, December). Design study methodology: Reflections from the trenches and the stacks. IEEE Transactions on Visualization and Computer Graphics, 18(12), 2431–2440. https://doi.org/10.1109/TVCG.2012.213
   PubMed Web of Science ®Google Scholar
• Sengers, P., Boehner, K., Mateas, M., & Gay, G. (2008, June). The disenchantment of affect. Personal and Ubiquitous Computing, 12(5), 347–358. https://doi.org/10.1007/s00779-007-0161-4
   Web of Science ®Google Scholar
• Sengers, P., & Gaver, B. (2006). Staying open to interpretation: Engaging multiple meanings in design and evaluation. In Proceedings of the ACM Conference on Designing Interactive Systems (DIS) (pp. 99–108). ACM.
   Google Scholar
• Seo, J., & Shneiderman, B. (2006, May). Knowledge discovery in high-dimensional data: Case studies and a user survey for the rank-by-feature framework. IEEE Transactions on Visualization and Computer Graphics, 12(3), 311–322. https://doi.org/10.1109/TVCG.2006.50
   PubMed Web of Science ®Google Scholar
• Shneiderman, B. (1996). The eyes have it: A task by data type taxonomy for information visualizations. In Proceedings of the IEEE Symposium on Visual Languages (pp. 336–343). IEEE.
   Google Scholar
• Shneiderman, B., & Plaisant, C. (2006). Strategies for evaluating information visualization tools: Multi-dimensional in-depth long-term case studies. In Proceedings of the 2006 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV) (pp. 1–7). ACM. https://doi.org/10.1145/1168149.1168158
   Google Scholar
• Smith, A., Kumar, V., Boyd-Graber, J., Seppi, K., & Findlater, L. (2018). Closing the loop: User-centered design and evaluation of a human-in-the-loop topic modeling system. In Proceedings of the ACM Conference on Intelligent User Interfaces (IUI) (pp. 293–304). ACM. https://doi.org/10.1145/3172944.3172965
   Google Scholar
• Smith, A., Lee, T. Y., Poursabzi-Sangdeh, F., Boyd-Graber, J., Elmqvist, N., & Findlater, L. (2017, January). Evaluating visual representations for topic understanding and their effects on manually generated topic labels. Transactions of the Association for Computational Linguistics, 5, 1–15. https://doi.org/10.1162/tacl_a_00042
   Google Scholar
• Springmann, U., Najock, D., Morgenroth, H., Schmid, H., Gotscharek, A., & Fink, F. (2014). OCR of historical printings of Latin texts: Problems, prospects, progress. In Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage (DATeCH) (pp. 71–75). ACM. https://doi.org/10.1145/2595188.2595205
   Google Scholar
• Steyvers, M., Smyth, P., Rosen-Zvi, M., & Griffiths, T. (2004). Probabilistic author-topic models for information discovery. In Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD) (pp. 306–315). ACM. https://doi.org/10.1145/1014052.1014087
   Google Scholar
• Strathern, M. (1997, July). ‘Improving ratings’: Audit in the British University system. European Review, 5(3), 305–321. https://doi.org/10.1017/S1062798700002660
   Google Scholar
• Sweeney, L. (2013). Discrimination in online ad delivery. Communications of the ACM, 56(5), 44–54. https://doi.org/10.1145/2447976.2447990
   Web of Science ®Google Scholar
• Taylor, A. S. (2009). Machine Intelligence. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (p. 2109–2118).
   Google Scholar
• Thompson, L., & Mimno, D. (2018). Authorless topic models: Biasing models away from known structure. In Proceedings of the International Conference on Computational Linguistics (COLING) (pp. 3903–3914). Association for Computational Linguistics.
   Google Scholar
• Underwood, T. (2012, April). What kinds of “topics” does topic modeling actually produce?
   Google Scholar
• Underwood, T. (2014). Theorizing research practices we forgot to theorize twenty years ago. Representations, 127(1), 64–72. https://doi.org/10.1525/rep.2014.127.1.64
   Google Scholar
• Underwood, T. (2019). Distant horizons. University of Chicago Press.
   Google Scholar
• Underwood, T., & Goldstone, A. (2013, September). List of stop words used in topic modeling journals, summer 2013. http://hdl.handle.net/2142/45709
   Google Scholar
• Wakefield, A. J., Murch, S. H., Anthony, A., Linnell, J., Casson, D. M., Malik, M., Berelowitz, M., Dhillon, A. P., Thomson, M. A., Harvey, P., Valentine, A., Davies, S. E., & Walker-Smith, J. A. (1998, February). RETRACTED: Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. The Lancet, 351(9103), 637–641. https://doi.org/10.1016/S0140-6736(97)11096-0
   PubMed Web of Science ®Google Scholar
• Yang, Q., Scuito, A., Zimmerman, J., Forlizzi, J., & Steinfeld, A. (2018). Investigating how experienced UX designers effectively work with machine learning. In Proceedings of the ACM Conference on Designing Interactive Systems (DIS) (pp. 585–596). ACM. https://doi.org/10.1145/3196709.3196730
   Google Scholar
• Zade, H., Drouhard, M., Chinh, B., Gan, L., & Aragon, C. (2018). Conceptualizing disagreement in qualitative coding. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI) (pp. 159: 1–159:11). ACM. https://doi.org/10.1145/3173574.3173733
   Google Scholar
• Zimmerman, J., & Forlizzi, J. (2014). Research through design in HCI. In J. S. Olson & W. A. Kellogg (Eds.), Ways of Knowing in HCI (pp. 167–189). Springer.
   Google Scholar