Supporting Crowd Workers in Ideation Tasks Through Information Gathering and Reflective Activity

References

• Abbas, T., Khan, V. J., & Markopoulos, P. (2020). Investigating the crowd’s creativity for creating on-demand IoT scenarios. International Journal of Human–Computer Interaction, 36(11), 1022–1049. https://doi.org/10.1080/10447318.2019.1709331
   Web of Science ®Google Scholar
• Acar, O. A. (2019). Motivations and solution appropriateness in crowdsourcing challenges for innovation. Research Policy, 48(8), 103716. https://doi.org/10.1016/j.respol.2018.11.010
   Web of Science ®Google Scholar
• Adams, R. S., Turns, J., & Atman, C. J. (2003). Educating effective engineering designers: The role of reflective practice. Design Studies, 24(3), 275–294. https://doi.org/10.1016/S0142-694X(02)00056-X
   Google Scholar
• Atman, C. J., Adams, R. S., Cardella, M. E., Turns, J., Mosborg, S., & Saleem, J. (2007). Engineering design processes: A comparison of students and expert practitioners. Journal of Engineering Education, 96(4), 359–379. https://doi.org/10.1002/j.2168-9830.2007.tb00945.x
   Web of Science ®Google Scholar
• Bayus, B. L. (2013). Crowdsourcing new product ideas over time: An analysis of the dell ideaStorm community. Management Science, 59(1), 226–244. https://doi.org/10.1287/mnsc.1120.1599
   Web of Science ®Google Scholar
• Bosch‐Sijtsema, P., & Bosch, J. (2015). User involvement throughout the innovation process in high-tech industries. Journal of Product Innovation Management, 32(5), 793–807. https://doi.org/10.1111/jpim.12233
   Web of Science ®Google Scholar
• Bursic, K. M., & Atman, C. J. (1997). Information gathering: A critical step for quality in the design process. Quality Management Journal, 4(4), 60–75. https://doi.org/10.1080/10686967.1998.11919148
   Google Scholar
• Buxton, B. (2010). Sketching user experience: Getting the design right and the right design. Morgan Kaufmann.
   Google Scholar
• Chan, J., Dang, S., & Dow, S. P. (2016). Improving crowd innovation with expert facilitation [Paper presentation]. Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 1223–1235). ACM. https://doi.org/10.1145/2818048.2820023
   Google Scholar
• Charyton, C., & Merrill, J. A. (2009). Assessing general creativity and creative engineering design in first year engineering students. Journal of Engineering Education, 98(2), 145–156. https://doi.org/10.1002/j.2168-9830.2009.tb01013.x
   Web of Science ®Google Scholar
• Chulvi, V., Sonseca, Á., Mulet, E., & Chakrabarti, A. (2012). Assessment of the relationships among design methods, design activities, and creativity. Journal of Mechanical Design, 134(11), 111004. https://doi.org/10.1115/1.4007362
   Web of Science ®Google Scholar
• Crilly, N. (2019). Creativity and fixation in the real world: A literature review of case study research. Design Studies, 64, 154–168. https://doi.org/10.1016/j.destud.2019.07.002
   Web of Science ®Google Scholar
• Dean, D. L., Hender, J. M., Rodgers, T. L., & Santanen, E. L. (2006). Identifying quality, novel, and creative ideas: Constructs and scales for idea evaluation. Journal of the Association for Information Systems, 7(10), 646–699. https://doi.org/10.17705/1jais.00106
   Web of Science ®Google Scholar
• Dhengre, S., Mathur, J., Oghazian, F., Tan, X., & McComb, C. (2020). Towards enhanced creativity in interface design through automated usability evaluation. In Proceedings of the 11th International Conference on Computational Creativity (ICCC’20) (pp. 366–369). Association for Computational Creativity.
   Google Scholar
• Diedrich, J., Benedek, M., Jauk, E., & Neubauer, A. C. (2015). Are creative ideas novel and useful? Psychology of Aesthetics, Creativity, and the Arts, 9(1), 35–40. https://doi.org/10.1037/a0038688
   Web of Science ®Google Scholar
• DiSalvo, B. (2016). Participatory design through a learning science lens. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 4459–4463). ACM. https://doi.org/10.1145/2858036.2858405
   Google Scholar
• Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012). Shepherding the crowd yields better work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (pp. 1013–1022). ACM. https://doi.org/10.1145/2145204.2145355
   Google Scholar
• George, T., Wiley, J., Koppel, R. H., & Storm, B. C. (2019). Constraining or constructive? The effects of examples on idea novelty. The Journal of Creative Behavior, 53(3), 396–403. https://doi.org/10.1002/jocb.178
   Web of Science ®Google Scholar
• Goldschmidt, G. (2015). Ubiquitous serendipity: Potential visual design stimuli are everywhere. In J. S. Gero (Ed.), Studying visual and spatial reasoning for design creativity (pp. 205–214). Springer.
   Google Scholar
• Goto, S., Ishida, T., & Lin, D. (2016). Understanding crowdsourcing workflow: Modeling and optimizing iterative and parallel processes. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 4(1), 52–58. https://ojs.aaai.org/index.php/HCOMP/article/view/13289
   Google Scholar
• Han, J., Forbes, H., & Schaefer, D. (2021). An exploration of how creativity, functionality, and aesthetics are related in design. Research in Engineering Design, 32(3), 289–307. https://doi.org/10.1007/s00163-021-00366-9
   Web of Science ®Google Scholar
• Hargadon, A. B. (2002). Brokering knowledge: Linking learning and innovation. Research in Organizational Behavior, 24, 41–85. https://doi.org/10.1016/S0191-3085(02)24003-4
   Web of Science ®Google Scholar
• Herring, S. R., Jones, B. R., & Bailey, B. P. (2009). Idea generation techniques among creative professionals. In Proceedings of the 42nd Hawaii International Conference on System Sciences (pp. 1–10). IEEE. https://doi.org/10.1109/HICSS.2009.241
   Google Scholar
• Heywood, J. (2005). Engineering education: Research and development in curriculum and instruction. IEEE Press.
   Google Scholar
• Hong, Y. C., & Choi, I. (2011). Three dimensions of reflective thinking in solving design problems: A conceptual model. Educational Technology Research and Development, 59(5), 687–710. https://doi.org/10.1007/s11423-011-9202-9
   Web of Science ®Google Scholar
• Hong, Y. C., & Choi, I. (2019). Relationship between student designers’ reflective thinking and their design performance in bioengineering project: Exploring reflection patterns between high and low performers. Educational Technology Research and Development, 67(2), 337–360. https://doi.org/10.1007/s11423-018-9618-6
   Google Scholar
• Hossain, M., & Islam, K. M. Z. (2015). Generating ideas on online platforms: A case study of “my starbucks idea. Arab Economic and Business Journal, 10(2), 102–111. https://doi.org/10.1016/j.aebj.2015.09.001
   Google Scholar
• Howe, J. (2006). The rise of crowdsourcing. Wired Magazine, 14(6), 1–4. http://www.wired.com/wired/archive/14.06/crowds_pr.html
   Google Scholar
• Johnson, H., & Carruthers, L. (2006). Supporting creative and reflective processes. International Journal of Human-Computer Studies, 64(10), 998–1030. https://doi.org/10.1016/j.ijhcs.2006.06.001
   Web of Science ®Google Scholar
• Kim, J., Cheng, J., & Bernstein, M. S. (2014). Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 745–755). ACM. https://doi.org/10.1145/2531602.2531638
   Google Scholar
• Kim, J., Sterman, S., Cohen, A. A. B., & Bernstein, M. S. (2017). Mechanical novel: Crowdsourcing complex work through reflection and revision. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (pp. 233–245). ACM. https://doi.org/10.1145/2998181.2998196
   Google Scholar
• Kittur, A. (2010). Crowdsourcing, collaboration and creativity. XRDS: Crossroads, the ACM Magazine for Students, 17(2), 22–26. https://doi.org/10.1145/1869086.1869096
   Google Scholar
• Liu, Q., Du, Q., Hong, Y., Fan, W., & Wu, S. (2020). User idea implementation in open innovation communities: Evidence from a new product development crowdsourcing community. Information Systems Journal, 30(5), 899–927. https://doi.org/10.1111/isj.12286
   Web of Science ®Google Scholar
• Liu, Y. C., Chakrabarti, A., & Bligh, T. (2003). Towards an ‘ideal’ approach for concept generation. Design Studies, 24(4), 341–355. https://doi.org/10.1016/S0142-694X(03)00003-6
   Google Scholar
• Lubart, T. I. (2001). Models of the creative process: Past, present and future. Creativity Research Journal, 13(3–4), 295–308. https://doi.org/10.1207/S15326934CRJ1334_07
   Web of Science ®Google Scholar
• Luo, L., & Toubia, O. (2015). Improving online idea generation platforms and customizing the task structure on the basis of consumers’ domain-specific knowledge. Journal of Marketing, 79(5), 100–114. https://doi.org/10.1509/jm.13.0212
   Web of Science ®Google Scholar
• Luther, K., Tolentino, J.-L., Wu, W., Pavel, A., Bailey, B. P., Agrawala, M., Hartmann, B., & Dow, S. P. (2015). Structuring, aggregating, and evaluating crowdsourced design critique. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 473–485). ACM. https://doi.org/10.1145/2675133.2675283
   Google Scholar
• Maher, M. L. (2012). Computational and collective creativity: Who’s being creative? Proceedings of the International Conference on Computational Creativity, 2012, 67–71.
   Google Scholar
• Mao, K., Capra, L., Harman, M., & Jia, Y. (2017). A survey of the use of crowdsourcing in software engineering. Journal of Systems and Software, 126, 57–84. https://doi.org/10.1016/j.jss.2016.09.015
   Web of Science ®Google Scholar
• Mumford, M. D., & McIntosh, T. (2017). Creative thinking processes: The past and the future. The Journal of Creative Behavior, 51(4), 317–322. https://doi.org/10.1002/jocb.197
   Web of Science ®Google Scholar
• Oppenlaender, J., Milland, K., Visuri, A., Ipeirotis, P., & Hosio, S. (2020). Creativity on paid crowdsourcing platforms. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–14). ACM. https://doi.org/10.1145/3313831.3376677
   Google Scholar
• Poetz, M. K., & Schreier, M. (2012). The value of crowdsourcing: Can users really compete with professionals in generating new product ideas? Journal of Product Innovation Management, 29(2), 245–256. https://doi.org/10.1111/j.1540-5885.2011.00893.x
   Web of Science ®Google Scholar
• Reiter-Palmon, R., & Illies, J. J. (2004). Leadership and creativity: Understanding leadership from a creative problem-solving perspective. The Leadership Quarterly, 15(1), 55–77. https://doi.org/10.1016/j.leaqua.2003.12.005
   Web of Science ®Google Scholar
• Ren, J., Nickerson, J. V., Mason, W., Sakamoto, Y., & Graber, B. (2014). Increasing the crowd’s capacity to create: How alternative generation affects the diversity, relevance and effectiveness of generated ads. Decision Support Systems, 65, 28–39. https://doi.org/10.1016/j.dss.2014.05.009
   Web of Science ®Google Scholar
• Restrepo, J., & Christiaans, H. (2004). Problem structuring and information access in design. Journal of Design Research, 4(2), 0–236. https://doi.org/10.1504/JDR.2004.009842
   Google Scholar
• Robertson, B. F., Walther, J., & Radcliffe, D. F. (2007). Creativity and the use of CAD tools: Lessons for engineering design education from industry. Journal of Mechanical Design, 129(7), 753–760. https://doi.org/10.1115/1.2722329
   Web of Science ®Google Scholar
• Rowland, G. (1993). Designing and instructional design. Educational Technology Research and Development, 41(1), 79–91. https://doi.org/10.1007/BF02297094
   Web of Science ®Google Scholar
• Sarkar, P., & Chakrabarti, A. (2011). Assessing design creativity. Design Studies, 32(4), 348–383. https://doi.org/10.1016/j.destud.2011.01.002
   Web of Science ®Google Scholar
• Schon, D. A. (1983). The reflective practitioner: How professionals think in action. Basic Books.
   Google Scholar
• Seitamaa-Hakkarainen, P., & Hakkarainen, K. (2001). Composition and construction in experts’ and novices’ weaving design. Design Studies, 22(1), 47–66. https://doi.org/10.1016/S0142-694X(99)00038-1
   Google Scholar
• Shai, O., Reich, Y., & Rubin, D. (2009). Creative conceptual design: Extending the scope by infused design. Computer-Aided Design, 41(3), 117–135. https://doi.org/10.1016/j.cad.2007.11.004
   Web of Science ®Google Scholar
• Sharmin, M., & Bailey, B. P. (2011). “I reflect to improve my design”: Investigating the role and process of reflection in creative design. In Proceedings of the 8th ACM Conference on Creativity and Cognition (pp. 389–390). ACM. https://doi.org/10.1145/2069618.2069710
   Google Scholar
• Sio, U. N., Kotovsky, K., & Cagan, J. (2015). Fixation or inspiration? A meta-analytic review of the role of examples on design processes. Design Studies, 39, 70–99. https://doi.org/10.1016/j.destud.2015.04.004
   Web of Science ®Google Scholar
• Smith, S. M., Ward, T. B., & Schumacher, J. S. (1993). Constraining effects of examples in a creative generation task. Memory & Cognition, 21(6), 837–845. https://doi.org/10.3758/bf03202751
   PubMed Web of Science ®Google Scholar
• Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research, and Evaluation, 9, 1–11. https://doi.org/10.7275/96jp-xz07
   Google Scholar
• Sun, L., Xiang, W., Chen, S., & Yang, Z. (2015). Collaborative sketching in crowdsourcing design: A new method for idea generation. International Journal of Technology and Design Education, 25(3), 409–427. https://doi.org/10.1007/s10798-014-9283-y
   Google Scholar
• Sun, L., Yang, Y., Chen, Y., Li, J., Wang, G., Tao, Y., & Yao, L. (2020). ShrinkyKit: 3D Printing shrinkable adaptations for everyday objects. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–7). https://doi.org/10.1145/3334480.3383034
   Google Scholar
• Sun, L., Zhou, Z., Wu, W., Zhang, Y., Zhang, R., & Xiang, W. (2020). Developing a toolkit for prototyping machine learning-empowered products. International Journal of Design, 14(2), 35–50.
   Google Scholar
• Sutcliffe, A., & Sawyer, P. (2013). Requirements elicitation: Towards the unknown unknowns. In 2013 21st IEEE International Requirements Engineering Conference (RE) (pp. 92–104). https://doi.org/10.1109/RE.2013.6636709
   Google Scholar
• Suzuki, R., Salehi, N., Lam, M. S., Marroquin, J. C., & Bernstein, M. S. (2016). Atelier: Repurposing expert crowdsourcing tasks as micro-internships. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 2645–2656). ACM. https://doi.org/10.1145/2858036.2858121
   Google Scholar
• Taramigkou, M., Apostolou, D., & Mentzas, G. (2017). Supporting creativity through the interactive exploratory search paradigm. International Journal of Human–Computer Interaction, 33(2), 94–114. https://doi.org/10.1080/10447318.2016.1220104
   Web of Science ®Google Scholar
• Tung, F. W., & Lin, S. D. (2020). Cocreating value with customers: A case study of a technology-based startup. In C. Stephanidis, A. Marcus, E. Rosenzweig, PL. P. Rau, A. Moallem, M. Rauterberg (Eds.), International conference on human-computer interaction - late breaking papers: User experience design and case studies (pp. 320–330). Springer.
   Google Scholar
• Valkenburg, R., & Dorst, K. (1998). The reflective practice of design teams. Design Studies, 19(3), 249–271. https://doi.org/10.1016/S0142-694X(98)00011-8
   Google Scholar
• Visser, F. S., Lugt, R. V. D., & Stappers, P. J. (2007). Sharing user experiences in the product innovation process: Participatory design needs participatory communication. Creativity and Innovation Management, 16(1), 35–45. https://doi.org/10.1111/j.1467-8691.2007.00414.x
   Google Scholar
• Wang, K., Nickerson, J., & Sakamoto, Y. (2018). Crowdsourced idea generation: The effect of exposure to an original idea. Creativity and Innovation Management, 27(2), 196–208. https://doi.org/10.1111/caim.12264
   Web of Science ®Google Scholar
• Wendell, K. B., Wright, C. G., & Paugh, P. (2017). Reflective decision-making in elementary students’ engineering design. Journal of Engineering Education, 106(3), 356–397. https://doi.org/10.1002/jee.20173
   Web of Science ®Google Scholar
• Woodman, R. W., Sawyer, J. E., & Griffin, R. W. (1993). Toward a theory of organizational creativity. The Academy of Management Review, 18(2), 293–321. https://doi.org/10.5465/amr.1993.3997517
   Web of Science ®Google Scholar
• Wooten, J. O., & Ulrich, K. T. (2017). Idea generation and the role of feedback: Evidence from field experiments with innovation tournaments. Production and Operations Management, 26(1), 80–99. https://doi.org/10.1111/poms.12613
   Web of Science ®Google Scholar
• Xu, A., Huang, S. W., & Bailey, B. (2014). Voyant: Generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 1433–1444). ACM. https://doi.org/10.1145/2531602.2531604
   Google Scholar
• Yen, Y. C. G., Dow, S. P., Gerber, E., & Bailey, B. P. (2017). Listen to others, listen to yourself: Combining feedback review and reflection to improve iterative design. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (pp. 158–170). ACM. https://doi.org/10.1145/3059454.3059468
   Google Scholar
• Yu, L., Kittur, A., & Kraut, R. E. (2014). Distributed analogical idea generation: Inventing with crowds. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems (pp. 1245–1254). ACM. https://doi.org/10.1145/2556288.2557371
   Google Scholar
• Yu, L., Kraut, R. E., & Kittur, A. (2016). Distributed analogical idea generation with multiple constraints [Paper presentation]. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 1236–1245). ACM. https://doi.org/10.1145/2818048.2835201
   Google Scholar
• Yu, L., & Nickerson, J. V. (2011). Cooks or cobblers?: Crowd creativity through combination [Paper presentation]. In Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems (pp. 1393–1402). ACM. https://doi.org/10.1145/1978942.1979147
   Google Scholar
• Yuan, A., Luther, K., Krause, M., Vennix, S. I., Dow, S. P., & Hartmann, B. (2016). Almost an expert: The effects of rubrics and expertise on perceived value of crowdsourced design critiques. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 1003–1015). ACM. https://doi.org/10.1145/2818048.2819953
   Google Scholar
• Zhao, M., & van der Hoek, A. (2015). A brief perspective on microtask crowdsourcing workflows for interface design [Paper presentation]. In 2015 IEEE/ACM 2nd International Workshop on Crowdsourcing in Software Engineering (pp. 45–46). IEEE. https://doi.org/10.1109/CSI-SE.2015.16
   Google Scholar
• Zhou, J., & George, J. M. (2003). Awakening employee creativity: The role of leader emotional intelligence. The Leadership Quarterly, 14(4–5), 545–568. https://doi.org/10.1016/S1048-9843(03)00051-1
   Web of Science ®Google Scholar