Human uncertainty in explicit user feedback and its impact on the comparative evaluations of accurate prediction and personalisation

References

• Amatriain, X., and J. Pujol. 2009. “Rate it Again: Increasing Recommendation Accuracy by User Re-rating.” Proceedings of the Third ACM Conference on Recommender Systems, 173–180. New York: ACM.
   Google Scholar
• Amatriain, X, J. M. Pujol, and N. Oliver. 2009. “I Like It…I Like it Not: Evaluating User Ratings Noise in Recommender Systems. Proceedings of UMAP, Trento, Italy, 247–258.
   Google Scholar
• Beel, J., M. Genzmehr, S. Langer, A. Nürnberger, and B. Gipp. 2013. “A Comparative Analysis of Offline and Online Evaluations and Discussion of Research Paper Recommender System Evaluation.” Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, Hong Kong, China, 7–14. RecSys '13.
   Google Scholar
• Bell, R., Y. Koren, and C. Volinsky. 2009. “Matrix Factorization Techniques for Recommender Systems.” Computer 42: 30–37.
   Web of Science ®Google Scholar
• Bevington, P., and D. K. Robinson. 2002. Data Reduction and Error Analysis for the Physical Sciences. 3rd ed. New York: McGraw-Hill Education.
   Google Scholar
• Bhojwani, S., S. Bhojwani, and D. Martin. 2009. Method and system employing a mobile device for soliciting user experience sharing. Patent US20090259499A1.
   Google Scholar
• Bishop, C. 2007. Pattern Recognition and Machine Learning. New York: Springer.
   Google Scholar
• Bobadilla, J., F. Ortega, A. Hernando, and A. Gutiérrez. 2013. “Recommender Systems Survey.” Knowledge-based systems 46: 109–132. doi: 10.1016/j.knosys.2013.03.012
   Web of Science ®Google Scholar
• Buffler, A., S. Allie, and F. Lubben. 2001. “The Development of First Year Physics Students' Ideas about Measurement in Terms of Point and Set Paradigms.” International Journal of Science Education 23 (11): 1137–1156. doi: 10.1080/09500690110039567
   Web of Science ®Google Scholar
• Chandrasekar, R., D. Slawson, M. Scott, D. Ayakkad, and D. Makoski. 2008. User feedback correlated to specific user interface or application features. Microsoft Corp. Patent US8191004B2.
   Google Scholar
• Devore, J., and R. Peck. 2017. Statistics: The Exploration and Analysis of Data. Boston: Brooks/Cole.
   Google Scholar
• Doya, K. 2007. Bayesian Brain: Probabilistic Approaches to Neural Coding. Cambridge, MA: MIT Press.
   Google Scholar
• Elliott, R., C. T. Fischer, and D. L. Rennie. 1999. “Evolving Guidelines for Publication of Qualitative Research Studies in Psychology and Related Fields.” British journal of clinical psychology 38 (3): 215–229. doi: 10.1348/014466599162782
   PubMed Web of Science ®Google Scholar
• Faisal, A. A., L. P. J. Selen, and D. M. Wolpert. 2008. “Noise in the Nervous System.” Nature Reviews Neuroscience 9: 292–303. doi: 10.1038/nrn2258
   PubMed Web of Science ®Google Scholar
• Friston, K. 2010. “The Free-energy Principle: A Unified Brain Theory?” Nature Reviews Neuroscience 11 (2): 127–138. doi: 10.1038/nrn2787
   PubMed Web of Science ®Google Scholar
• Friston, K., P. Schwartenbeck, T. FitzGerald, M. Moutoussis, T. Behrens, and R. J. Dolan. 2013. “The Anatomy of Choice: Active Inference and Agency.” Frontiers in Human Neuroscience 7:58. doi: 10.3389/fnhum.2013.00598
   PubMed Web of Science ®Google Scholar
• Grabe, M. 2011. Grundriss der generalisierten gauß'schen fehlerrechnung. Berlin, Heidelberg: Springer Berlin Heidelberg.
   Google Scholar
• Grcar, M., D. MladenicBlaz, and F. M. Grobelnik. 2005. “Data Sparsity Issues in the Collaborative Filtering Framework.” In Advances in Web Mining and Web Usage Analysis, edited by WebKDD, 58–76. Chicago, IL: ACM.
   Google Scholar
• Henze, N. 2013. Stochastik für einsteiger. Berlin, Heidelberg: Springer Spektrum.
   Google Scholar
• Herlocker, J. L., J. A. Konstan, L. G. Terveen, and J. T. Riedl. 2004. “Evaluating Collaborative Filtering Recommender Systems.” ACM Transactions on Information Systems 22 (1): 5–53. doi: 10.1145/963770.963772
   Web of Science ®Google Scholar
• Hill, W., L. Stead, M. Rosenstein, and G. Furnas. 1995. “Recommending and Evaluating Choices in a Virtual Community of Use.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Denver, CO, USA, 194–201. CHI '95.
   Google Scholar
• Hu, L., A. Sun, and Y. Liu. 2014. “Your Neighbors Affect Your Ratings: On Geographical Neighborhood Influence to Rating Prediction.” Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, 345–354. New York, NY: ACM.
   Google Scholar
• Intraub, H. 1990. “Presentation Rate and the Representation of Briefly Glimpsed Pictures in Memory.” Journal of Experimental Psychology 6 (1): 1–11.
   Google Scholar
• Jasberg, K. 2017. “Assessment of Prediction Techniques: The Impact of Human Uncertainty.” Proceedings of WISE. Cham: Springer.
   Google Scholar
• Jasberg, K., and S. Sizov. 2017. “The Magic Barrier Revisited: Accessing Natural Limitations of Recommender Assessment.” In RecSys '17, 56–64. New York, NY: ACM.
   Google Scholar
• Jasberg, K., and S. Sizov. 2018. Dealing with uncertainties in user feedback: Strategies between denying and accepting. ArXiv:1802.05895.
   Google Scholar
• Jawaheer, G., M. Szomszor, and P. Kostkova. 2010. “Comparison of Implicit and Explicit Feedback from an Online Music Recommendation Service.” Proceedings of the 1st International Workshop on Information Heterogeneity and Fusion in Recommender, 47–51. HetRec '10. New York, NY: ACM.
   Google Scholar
• JCGM. 2008a. Guide to the Expression of Uncertainty in Measurement. Sèvres: BIPM.
   Google Scholar
• JCGM. 2008b. Supplement 1 to the Gum - Propagation of Distributions Using a Monte Carlo Method. Sèvres: BIPM.
   Google Scholar
• Jung, S., J. L. Herlocker, and J. Webster. 2007. “Click Data as Implicit Relevance Feedback in Web Search.” Information Processing and Management 43 (3): 791–807. doi: 10.1016/j.ipm.2006.07.021
   Web of Science ®Google Scholar
• Knijnenburg, B. P., M. C. Willemsen, Z. Gantner, H. Soncu, and C. Newell. 2012. “Explaining the user experience of recommender systems.” User Modeling and User-Adapted Interaction 22: 441–504. doi: 10.1007/s11257-011-9118-4
   Web of Science ®Google Scholar
• Koren, Y. 2008. “Factorization Meets the Neighborhood: A Multifaceted Collaborative Filtering Model.” Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 426–434. New York: ACM.
   Google Scholar
• Koren, Y., and J. Sill. 2011. “Ordrec: An Ordinal Model for Predicting Personalized Item Rating Distributions.” Proceedings of ACM RecSys. New York: ACM.
   Google Scholar
• Ku, H. 1966. “Notes on the Use of Propagation of Error Formulas.” Journal of Research of the National Bureau of Standards 70 (4): 263.
   Google Scholar
• Kubat, M. 2015. An Introduction to Machine Learning. Berlin: Springer.
   Google Scholar
• Lee, L. 1999. “Measures of Distributional Similarity.” Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: Association for Computational Linguistics.
   Google Scholar
• Ma, W. J., J. M Beck, P. E Latham, and A. Pouget. 2006. “Bayesian Inference with Probabilistic Population Codes.” Nature Neuroscience 9 (11): 1432–1438. doi: 10.1038/nn1790
   PubMed Web of Science ®Google Scholar
• Manning, C. D., P. Raghavan, and H. Schuetze. 2008. Introduction to Information Retrieval. New York: Cambridge University Press.
   Google Scholar
• McNee, S., J. Riedl, and J. A. Konstan. 2006. “Being Accurate is not Enough: How Accuracy Metrics Have Hurt Recommender Systems.” In CHI'06 Extended Abstracts on Human Factors in Computing Systems. New York: ACM.
   Google Scholar
• Morris, M. R., and E. Horvitz. 2007. “Searchtogether: An Interface for Collaborative Web Search.” Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, 3–12. New York: ACM.
   Google Scholar
• Netflix. 2016. The netflix prize rules.
   Google Scholar
• Peska, L. 2016. Using the context of user feedback in recommender systems. arXiv:161204978.
   Google Scholar
• Rashid, A. M., I. Albert, D. Cosley, S. K. Lam, S. M. McNee, J. A. Konstan, and J. Riedl. 2002. Getting to know you: learning new user preferences in recommender systems. Proceedings of the 7th international conference on Intelligent user interfaces, 127–134. New York: ACM.
   Google Scholar
• Ricci, F., L. Rokach, and B. Shapira. 2015. Recommender Systems Handbook. New York: Springer.
   Google Scholar
• Said, A. 2012. “Users and Noise: The Magic Barrier of Recommender Systems.” Proceedings of UMAP, 237–248. Berlin: Springer.
   Google Scholar
• Said, A., and A. Bellogín. 2018. “Coherence and Inconsistencies in Rating Behavior: Estimating the Magic Barrier of Recommender Systems.” User Modeling and User-Adapted Interaction 28: 97–125. doi: 10.1007/s11257-018-9202-0
   Web of Science ®Google Scholar
• Salakhutdinov, R., and A. Mnih. 2008. “Probabilistic Matrix Factorization.” In Advances in Neural Information Processing Systems, Vancouver, 1257–1264.
   Google Scholar
• Settles, B. 2009. “Active Learning Literature Survey.” University of Wisconsin–Madison. Computer Sciences Technical Report 1648.
   Google Scholar
• Sizov, S. 2017. “Mining Ordinal Data Under Human Response Uncertainty.” Proceedings of the International Conference on Web Intelligence, 869–876. New York, NY: ACM.
   Google Scholar
• Stinken, L. 2017. Ich hoffe du weißt das zu schätzen?: eine erhebung der schätzkompetenz in der sekundarstufe i. Berlin: Logos Verlag.
   Google Scholar
• Stumpf, S., V. Rajaram, L. Li, M. Burnett, T. Dietterich, E. Sullivan, R. Drummond, and J. Herlocker. 2007. “Toward Harnessing user Feedback for Machine Learning.” Proceedings of the 12th International Conference on Intelligent User Interfaces. New York, NY: ACM.
   Google Scholar
• Taylor, J. 1997. Introduction to Error Analysis, the Study of Uncertainties in Physical Measurements. Sausalito, CA: University Science Books.
   Google Scholar
• Walcher, W. 2006. Praktikum der physik. Wiesbaden: Vieweg + Teubner.
   Google Scholar
• Zanker, M., A. Felfernig, and G. Friedrich. 2010. Recommender Systems: An Introduction. New York: Cambridge University Press.
   Google Scholar
• Zhang, H. R., F. Min, Y. X. Wu, Z. L. Fu, and L. Gao. 2018. “Magic Barrier Estimation Models for Recommended Systems under Normal Distribution.” Applied Intelligence 48: 4678–4693. doi: 10.1007/s10489-018-1237-8
   Web of Science ®Google Scholar
• Zhao, G., X. Qian, and C. Kang. 2017. “Service Rating Prediction by Exploring Social Mobile Users' Geographic Locations.” IEEE Transactions on Big Data 3 (1): 67–78. doi: 10.1109/TBDATA.2016.2552541
   Web of Science ®Google Scholar
• Zhao, G., X. Qian, X. Lei, and T. Mei. 2016. “Service Quality Evaluation by Exploring Social Users' Contextual Information.” IEEE Transactions on Knowledge and Data Engineering 28 (12): 3382–3391.
   Web of Science ®Google Scholar
• Ziegler, C. N., S. M. McNee, J. A. Konstan, and G. Lausen. 2005. “Improving Recommendation Lists through Topic Diversification.” Proceedings of the 14th International Conference on World Wide Web, 22–32. New York, NY: ACM.
   Google Scholar