https://orcid.org/0000-0002-9575-5080
![Sample our Language & Literature journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/5941/TOC-Subject-Sample-2021-Language-Literature.jpg"})
ABSTRACT
Mass adoption of neural machine translation (NMT) tools in the translation workflow has exerted a significant impact on the language services industry over the last decade. There are claims that with the advent of NMT, automated translation has reached human parity for translating news (see, e.g. Popel et al. 2020). Moreover, some machine translation (MT) research has already been done in the context of literary texts. In this paper, we share the results of a pilot study carried out with two groups (a pre-course group and post-course group) of MA-level students participating in a course that involved translating culture-bound texts. The students’ role was to post-edit and evaluate two machine-translated stories (Polish legends), marking their comprehensibility and accuracy. We discuss the lessons learnt during this pilot study, the critical errors detected by the students and their perceptions of the end products and the experiment itself. We report noticeable differences found between the pre-course group and the post-course group in terms of language awareness and the speed and quality of their post-editing (PE) performance. Our results also show that the task of post-editing culture-bound texts offers students a unique and enjoyable setting, enabling them to assess translation technology and hone their translation skills at the same time.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Notes
1. BLEU (Bilingual Evaluation Understudy Score) is the oldest and most popular automated MT quality metric, proposed by Papineni et al. (Citation2002). In some specialised contexts, BLEU scores may reach very high values (e.g. 62.5 in an EU law document as reported by Łoboda Citation2021), which indicates they could be considered appropriate for MT processes.
2. In this paper we use consistently the term ‘translation unit’, or TU, for a chunk of text processed in a translation environment. In the field of natural language processing a TU is also often referred to as a ‘segment’ or a ‘sequence’.
3. Cf. also an interesting study on MT vs. creativity by Guerberof Arenas and Toral (Citation2022).
4. For more detailed descriptions of projects 2 and 3, and their advantages from the point of view of translation trainees, see Mastela (Citation2022) and Mastela (Citation2023, forthcoming). For an informed analysis of another authentic student translation project in which a style guide for translating folk tales was developed, see Mastela (Citation2020a).
5. The authors are aware that the error typology has been constantly evolving, and the so-called MQM (Multidimensional Quality Metrics) framework has recently been used as the basis for the draft standard ISO/DIS 5060:Citation2022. However, there is an ongoing discussion on the scope of application for this standard.
6. Technically, it is a deep neural network-based classifier whose usefulness in most contexts and language combinations is not guaranteed at this point. The tool was developed with ‘large scale datasets of MT outputs and their postedits’ (Tamchyna Citation2020, 294). Tamchyna (Citation2020) explains that the selection of MT system is static, i.e. text- rather than TU-based. Only by comparing the result in Memsource could we conclude that Google Translate NMT was chosen by the system for both texts, though this was not indicated in the tool in any way. Moreover, the user had no control over the MT engine selection process.
7. Calculated with tokenisation of the TUs.
8. According to the revised MQM 2.0 error typology, issues such as grammar, punctuation, spelling and the character of being unintelligible are labelled as ‘linguistic conventions’, whereas register and various stylistic features are labelled more generally as ‘style’ (cf. MQM Core Typology Citation2023).
9. See the definition of ‘warlock’ at www.lexico.com, for example.
10. We did not intend to measure the cognitive effort, but focused instead on the two other types distinguished by Krings (Citation2001) and explored by O’Brien (Citation2007), namely the technical and temporal effort. The pilot study was performed when pandemic regulations were binding: students had to wear face masks at all times during the session or had to use remote communication platforms. Besides, at the time of the study, the pre-course group was not familiar with more advanced CAT tools or with MT technologies. Therefore, we used bilingual RTF files prepared in Memsource, which could be edited by the students and contained a separate column for comments. The files could be then imported back to Memsource. In a planned larger study, we intend to measure the technical effort in MTPE of culture-bound texts in terms of edits (deletions, additions and substitutions) using the keylogging software available in our department and to examine more fine-grained error categories. This was outside of the scope of the pilot study where we mostly wanted to examine the performance of Group 1 and Group 2 in a more general way.
11. Such as ‘Wlazł ci dziadyga za piec’ [There you’ve got the codger sneak behind the stove’], which was machine-translated as ‘He put your grandfather behind the stove’.
12. Only one person noticed a problem in this respect.
13. For example, ‘dziad’ and its augmentative forms ‘dziadyga’, ‘dziadzisko’, which are colloquial and scornful names for ‘(old) man’ or ‘beggar’, were machine-translated as either ‘grandfather’ or ‘grandpa’ with absolutely no allusion to beggary.
14. However, this student asked for additional time and finally managed to correct 34 TUs, focusing on more important terminological and grammatical mistakes.