Man or machine? Comparing the difficulty of human translation versus neural machine translation post-editing

References

• Ai, M., Li, K., Liu, S., & Lin, D. K. (2013). Balanced incomplete latin square designs. Journal of Statistical Planning and Inference, 143(9), 1575–1582. https://doi.org/10.1016/j.jspi.2013.05.001
   Web of Science ®Google Scholar
• Balling, L. (2008). A brief introduction to regression designs and mixed-effects modelling by a recent convert. In A. L. Jakobsen, S. Göpferich, & I. M. Mees (Eds.), Looking at eyes (pp. 175–191). Samfunslitterattur.
   Google Scholar
• Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
   Web of Science ®Google Scholar
• Cadwell, P., O’Brien, S., & Teixeira, C. S. C. (2018). Resistance and accommodation: Factors for the (non-) adoption of machine translation among professional translators. Perspectives, 26(3), 301–321. https://doi.org/10.1080/0907676X.2017.1337210
   Web of Science ®Google Scholar
• Carl, M. (2012). Translog-II: A program for recording user activity data for empirical reading and writing research. Paper presented at the eighth international conference on language resources and evaluation (LREC12), Istanbul.
   Google Scholar
• Castilho, S., Moorkens J., Gaspari F., Calixto I., Tinsley J., Way A. (2017). Is neural machine translation the new state of the art? The Prague Bulletin of Mathematical Linguistics, 108(1), 109–120. https://doi.org/10.1515/pralin-2017-0013
   Google Scholar
• Castilho, S., Moorkens J., Gaspari F., Sennrich R., Way A. (2018). Evaluating MT for massive open online courses. Machine Translation, 32(3), 255–278. https://doi.org/10.1007/s10590-018-9221-y
   Web of Science ®Google Scholar
• Corder, G. W., & Foreman, D. I. (2009). Nonparametric statistics for non-statisticians: A step-by-step approach. Wiley.
   Google Scholar
• Da Silva, I. A. L., Alves, F., Schmaltz, M., Pagano, A., Wong, D., Chao, L., Leal, A. L. V., Quaresma, P., Garcia, C., & da Silva, G. E. (2017). Translation, post-editing and directionality: A study of effort in the Chinese-Portuguese language pair. In A. L. Jakobsen, & B. Mesa-Lao (Eds.), Translation in transition (pp. 107–134). John Benjamins.
   Google Scholar
• Daems, J., Vandepitte, S., Hartsuiker, R. J., & Macken, L. (2017). Translation methods and experience: A comparative analysis of human translation and post-editing with students and professional translators. Meta, 62(2), 245–270. https://doi.org/10.7202/1041023ar
   Web of Science ®Google Scholar
• Fox, J., Weisberg, S., Price, B., Friendly, M., Hong, J., Andersen, R., Firth, D., Taylor, S., & R Core Team. (2020). Effects: Effect displays for linear, generalized linear, and other models. https://cran.r-project.org/web/packages/effects/index.html
   Google Scholar
• Green, S., Heer, J., & Manning, C. D. (2013). The efficacy of human post-editing for language translation. In W. E. Mackay (Ed.), CHI ‘13: Proceedings of the SIGCHI conference on human factors in computing systems (pp. 439–448). Association for Computing Machinery.
   Google Scholar
• Herbig, N., Pal, S., Vela, M., Krüger, A., & van Genabith, J. (2019). Multi-modal indicators for estimating perceived cognitive load in post-editing of machine translation. Machine Translation, 33(1-2), 91–115. https://doi.org/10.1007/s10590-019-09227-8
   Web of Science ®Google Scholar
• Hox, J., & McNeish, D. (2020). Small samples in multilevel modeling. In R. Van de Schoot, & M. Miocević (Eds.), Small sample size solutions (pp. 215–225). Taylor & Francis.
   Google Scholar
• Hvelplund, K. T. (2014). Eye tracking and the translation process: Reflections on the analysis and interpretation of eye tracking data. MonTI Special Issue – Minding Translation, 1, 201–223. http://dx.doi.org/10.6035/MonTI.2014.ne1.6
   Google Scholar
• Jia, Y., Carl, M., & Wang, X. (2019). How does the post-editing of neural machine translation compare with from-scratch translation? A product and process study. The Journal of Specialised Translation, 31, 60–86.
   Google Scholar
• Jia, Y., & Zheng, B. (2022). The interaction effect between source text complexity and machine translation quality on the task difficulty of NMT post-editing from English to Chinese: A multi-method study. Across Languages and Cultures, 23(1), 36–55. https://doi.org/10.1556/084.2022.00120
   Web of Science ®Google Scholar
• Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4), 329–354. https://doi.org/10.1037/0033-295X.87.4.329
   PubMed Web of Science ®Google Scholar
• Koehn, P. (2010). Statistical machine translation. Cambridge University Press.
   Google Scholar
• Koglin, A. (2015). An empirical investigation of cognitive effort required to post-edit machine translated metaphors compared to the translation of metaphors. Translation & Interpreting, 7(1), 126–141.
   Google Scholar
• Krings, H. P. (2001). G. S. Koby (Trans.), Repairing texts: Empirical investigations of machine translation post-editing processes. Kent State University Press.
   Google Scholar
• Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017). Lmertest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13), 1–26. https://doi.org/10.18637/jss.v082.i13
   Web of Science ®Google Scholar
• LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815–852. https://doi.org/10.1177/1094428106296642
   Web of Science ®Google Scholar
• Lenth, R. V., Buerkner, P., Herve, M., Jung, M., Love, J., Miguez, F., Riebl, H., & Singmann, H. (2021). Emmeans: Estimated Marginal Means, aka Least-Squares Means. https://cran.r-project.org/web/packages/emmeans/index.html
   Google Scholar
• Lommel, A., Burchardt, A., Popović, M., Harris, K., Avramidis, E., & Uszkoreit, H. (2014). Using a new analytic measure for the annotation and analysis of MT errors on real data. In Proceedings of the 17th annual conference of the european association for machine translation (pp. 165–172). European Association for Machine Translation.
   Google Scholar
• McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press.
   Google Scholar
• Mesa-Lao, B. (2014). Gaze behaviour on source texts: An exploratory study comparing translation and post-editing. In L. W. Balling, M. Carl, M. Simard, & L. Specia (Eds.), Post-editing of machine translation (pp. 219–245). Cambridge Scholars Publishing.
   Google Scholar
• Meshkati, N. (1988). Toward development of a cohesive model of workload. In P. A. Hancock, & N. Meshkati (Eds.), Human mental workload (pp. 305–314). North-Holland.
   Google Scholar
• O’Brien, S. (2017). Machine translation and cognition. In J. W. Schwieter, & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 313–331). Wiley Blackwell.
   Google Scholar
• Paas, F. G. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84(4), 429–434. https://doi.org/10.1037/0022-0663.84.4.429
   Web of Science ®Google Scholar
• R Core Team. (2020). R: A language and environment for statistical computing. https://www.R-project.org
   Google Scholar
• Shterionov, D., Superbo, R., Nagle, P., Casanellas, L., O’Dowd, T., & Way, A. (2018). Human versus automatic quality evaluation of NMT and PBSMT. Machine Translation, 32(3), 217–235. https://doi.org/10.1007/s10590-018-9220-z
   Web of Science ®Google Scholar
• Sun, S. (2019). Measuring difficulty in translation and post-editing: A review. In D. Li, V. L. C. Lei, & Y. He (Eds.), Researching cognitive processes of translation (pp. 139–168). Springer.
   Google Scholar
• Sun, S., & Shreve, G. M. (2014). Measuring translation difficulty: An empirical study. Target. International Journal of Translation Studies, 26(1), 98–127. https://doi.org/10.1075/target.26.1.04sun
   Web of Science ®Google Scholar
• TAUS. (2011). Dynamic quality evaluation framework. https://www.taus.net/component/rsfiles/download-file/files?path=Reports%2FFree+Reports%2Ftausdynamicquality.pdf
   Google Scholar
• TAUS. (2013). Pricing machine translation post-editing guidelines. https://www.taus.net/academy/best-practices/postedit-best-practices/pricing-machinetranslation-post-editing-guidelines
   Google Scholar
• TAUS. (2016). MT post-editing guidelines. https://www.taus.net/academy/best-practices/postedit-best-practices/machine-translation-post-editing-guidelines
   Google Scholar
• TAUS. (2019). Keynotes summer 2019. https://info.taus.net/download-taus-keynotes-summer-2019
   Google Scholar
• Vieira, L. N. (2016). Cognitive effort in post-editing of machine translation: Evidence from eye movements, subjective ratings, and think-aloud protocols. [Doctoral dissertation]. Newcastle University. https://research-information.bris.ac.uk/en/publications/cognitive-effort-in-post-editing-of-machine-translation-evidence-
   Google Scholar
• Vieira, L. N. (2020). Post-editing of machine translation. In M. O’Hagan (Ed.), The routledge handbook of translation and technology (pp. 319–335). Routledge.
   Google Scholar
• Wang, X., Wang, T., Muñoz Martín, R., & Jia, Y. (2021). Investigating usability in postediting neural machine translation: Evidence from translation trainees’ self-perception and performance. Across Languages and Cultures, 22(1), 100–123. https://doi.org/10.1556/084.2021.00006
   Web of Science ®Google Scholar
• Zuur, A. F., Ieno, E. N., Walker, N., Saveliev, A. A., & Smith, G. M. (2009). Mixed effects models and extensions in ecology with R. Springer.
   Google Scholar