Assessing the learning of knowledge work competence in higher education – cross-cultural translation and adaptation of the Collaborative Knowledge Practices Questionnaire

ABSTRACT

The Collaborative Knowledge Practices Questionnaire (CKP) is an instrument designed to measure the learning of knowledge-work competence in education. The focus is on qualities of knowledge work which can be learned and taught in multiple educational settings and which may be especially important for courses with collaborative assignments. The original instrument was theoretically based on the knowledge-creation metaphor of learning. The instrument has been validated in Finnish based on student responses from a large number of higher education courses. The validation of the instrument resulted in seven scales relating to different aspects of interdisciplinary, collaborative development of knowledge-objects using digital technology. This study aimed to cross-culturally translate and adapt the original instrument into English and perform an exploratory structural equation modelling (ESEM) analysis in order to investigate whether the same factorial solution of the instrument also works in English in higher education courses in international settings. The original instrument was translated according to established guidelines for cross-cultural adaptation of self-report measures. The translated version has been tested in courses in medical education, online teaching and problem solving. The results provided evidence that the latent factor model found in the original instrument provided a good fit also for the adapted questionnaire.

KEYWORDS:

• Questionnaire
• collaborative knowledge-creation
• higher education

Background

There is an increased interest in generic, subject-independent skills in higher education and especially such skills that are important during work life and which could be better addressed already during education (Hyytinen, Toom, and Postareff 2018). Today’s students will have to tackle jobs that are profoundly different from existing ones involving requirements on developing competence in various practices which are typically not taught in higher education (European Union 2010). It has been pointed out that educational systems need to equip young people with new skills and competence adapted to the developments in society and economy which have come to view knowledge as the main asset (Ananiadou and Claro 2009). There is also a mismatch between formal education and the challenges of society regarding fostering skills for innovation (Clarke and Clarke 2009; Cobo 2013; Klusek and Bernstein 2006). In order to manage the changing requirements in the society and in the work life, new type of skills are therefore needed, such as collaborative learning, cultural awareness, self-leadership and flexibility, besides the ‘traditional’ work life skills, such as teamwork and social skills (Ilomäki, Lakkala, and Kosonen 2013). In addition, the transforming labour market generates new professions and changes in existing professions. For instance, outsourcing and entrepreneurship require competencies, which typically are not taught in higher education (European Union 2010). Most such skills – often referred to as ‘21st century skills and competencies’ (Voogt and Roblin 2012) to indicate that they are related to emerging models of economic and social development – can either be supported or enhanced by information technology (Ananiadou and Claro 2009). Formal education is also expected to support students in acquiring abilities to use technologies for collaboration and innovation, but research indicates that pedagogical changes have not actualised as expected, and this is a concern for both higher education and upper secondary schools (Clarke and Clarke 2009; Klusek and Bernstein 2006; Tynjälä 2008). While there is an emerging theoretical trend to highlight knowledge creation practices as a basis for understanding modern knowledge work (Knorr Cetina 2001), prevalent pedagogical methods and practices focus on content learning rather than on fostering the learning of professional knowledge work practices (Muukkonen et al. 2010). Students are reported to leave higher education with underdeveloped abilities to collaborate, manage their work processes, use computers, or solve open-ended problems (Arum and Roksa 2011; The National Commission on Accountability in Higher Education 2005; Tynjälä 2008).

Previous research has addressed generic skills related to collaboration and group work in various ways by, e.g. asking students to assess their skills and attitudes to collaboration, by asking them to assess activities influencing group work and by asking them to self-evaluate their skills at the end of study programmes (Muukkonen et al. 2019). Badcock and colleagues (Badcock, Pattison, and Harris 2010) found that university students’ grade point averages (GPAs) were significantly related to four generic skills – critical thinking, interprofessional understandings, problem solving and written communication. But while a grade point average provides some measure of students’ generic skills levels, the authors conclude that GPA should be considered an imperfect indicator of levels of skills attainment. More precise instruments are needed for addressing how the skills develop and for explicating which activities are involved in such development.

For example, in the field of engineering, little is known about how students perceive generic skills in the context of their discipline and there has been a lack of instruments for assessing students’ perceptions of these skills (Chan, Zhao, and Luk 2017). A questionnaire intended for self-assessment of first-year engineering students’ perceived levels of competency in generic skills within an engineering context was therefore developed by Chan and colleagues; the Generic Skills Perception Questionnaire was made up of eight scales and 35 items and exploratory and confirmatory factor analyses found the questionnaire to be a reliable and valid instrument for the target group. Similarly, but in a business school context, de la Harpe and colleagues described the process of identifying a set of generic skills – communication, computer literacy, information literacy, team working and decision-making – and how these were implemented in teaching (de la Harpe, Radloff, and Wyber 2010). The authors argued that universities may have to change their curricula to ensure quality and better meet the requirements of workplaces and, therefore, measures of effectiveness such as student questionnaires addressing such changes need to be developed (ibid.).

In a study by Crebert and colleagues, university graduates from three different study areas with work placement as a part of their studies perceived teamwork, being given responsibility and collaborative learning, as the most important factors for effective learning not only at university but during work placement and employment (Crebert et al. 2004). Crebert et al. pointed out the contrast between learning at universities where emphasis is on the individual student’s needs whereas learning in the workplace has emphasis on the organisation’s or the client’s needs. While much research has focused on investigating the individual dimensions of generic skills, collective dimensions such as collaborative idea generation or knowledge production and skills associated with shared knowledge advancement have not been addressed to the same extent by existing questionnaire instruments (Muukkonen et al. 2017). In addition, the existing instruments are often multiple-choice tests focusing on measuring only some aspects of generic skills, for example critical thinking (Hyytinen et al. 2015). They also lack the link to the authentic competence learning situations and pedagogical practices utilised in courses.

To address the above-mentioned shortcomings in previous instruments, Muukkonen et al. have developed the Collaborative Knowledge Practices Questionnaire (CKP) which is an instrument designed to evaluate self-assessed competence development during courses that specifically use group work or collaborative knowledge creation assignments as instructional approaches (Muukkonen et al. 2019, 2017). The instrument is not tied to a specific discipline and has a theoretical basis in socio-cultural theories of learning and particularly the knowledge creation metaphor and the Trialogical learning approach (Muukkonen et al. 2019; Paavola and Hakkarainen 2005). While many learning theories have emphasised how individual learners assimilate prevailing knowledge, or, how learners adapt to existing cultural and communal practices (Sfard 1998), Trialogical learning theory adds the object-centred approach to learning and development of expertise, typical of knowledge work. This approach draws attention to the presence of artefacts, practices, and products – ‘objects’ – and shared work on advancing them.

Such objects are viewed as being able to mediate knowledge advancement in several ways, e.g. by externalising ideas, supporting collaborative work practices and encouraging reflection on these (Hakkarainen 2009; Paavola and Hakkarainen 2005; Paavola et al. 2011) and important for knowledge creation is, therefore, to set up collaborative work practices around the objects in ways that promote knowledge advancement. A set of design principles have been developed to characterise the main features of trialogical learning in order to promote them theoretically in pedagogical practices and for the design of educational technology (Paavola et al. 2011). The design principles are the following: (1) Organising activities around shared ‘objects’; (2) Supporting integration of personal and collective agency and work through 
developing shared objects; (3) Emphasising development and creativity in working on shared objects through transformations and reflection; (4) Fostering long-term processes of knowledge advancement with shared objects (artefacts and practices); (5) Promoting cross-fertilisation of various knowledge practices and artefacts across communities and institutions and (6) Providing flexible tools for developing artefacts and practices (Paavola et al. 2011). The set of principles provided a starting point for the development of the CKP questionnaire (Muukkonen et al. 2019). There was however not a one-to-one correspondence between the design principles and the scales of the CKP; while the former were developed for designing and developing educational practices, the exact same structure was not found in the items addressing students’ learning experiences.

Whereas many instruments attempt to measure the level of competences in general, the CKP questionnaire addresses the self-evaluated learning of knowledge work practices in context, typically at the end of a course. The instrument focuses on a number of different aspects relating to course-related learning of collaborative practices and consists of seven scales, namely (1) Learning to collaborate on shared objects, (2) Integrating individual and collaborative working, (3) Development through feedback, (4) Persistent development of knowledge objects, (5) Understanding various disciplines and practices, (6) Interdisciplinary collaboration and communication and (7) Learning to exploit technology. The construct validity of the instrument has been investigated and measurement invariance was obtained in two disciplinary fields, media engineering and life sciences. The instrument has, however, not been systematically translated and tested in English.

Cross-cultural translation, adaptation and factorial evaluation

Numerous papers have addressed the challenges associated with translating instruments from one language and cultural context to another one and have proposed procedures and recommendations to ensure quality of the instruments (Beaton et al. 2000, 2007; Borsa, Damásio, and Bandeira 2012; Chen and Boore 2010; Guillemin, Bombardier, and Beaton 1993; Morales 2001; Schmidt, Power, and Bullinger 2002; van Widenfelt et al. 2005). Translation is viewed as the first step, but when an instrument is intended to be used in a different context, cultural, idiomatic, linguistic and contextual aspects need to be considered when adapting it to the new context (Borsa, Damásio, and Bandeira 2012). The literature suggests a number of essential stages for performing the cross-cultural translation and adaptation: (1) Translation of the instrument from the source language into the target language involving at least two independent, bilingual translators (Beaton et al. 2007; Borsa, Damásio, and Bandeira 2012) to minimise risks for biases and to detect ambiguous wording in the original language or discrepancies in translations. (2) Creating a synthesis during which semantic, idiomatic, conceptual, experiential, linguistic and contextual aspects are compared in the different translations. (3) Back-translation of the synthesised version into the original language by two new translators who are blind to the original version. This stage is a kind of validity checking ensuring that the translation reflects the item content of the original version and may detect unclear wordings in the translations (Beaton et al. 2007). (4) Next, the translations should be reviewed by an expert committee to develop a pre-final version of the questionnaire including instructions and scoring documentation (Beaton et al. 2007). The original developers should be involved as well as all translators. Decisions need to made regarding semantic equivalence (checking that words mean the same thing), idiomatic equivalence (colloquialisms or idioms may need to be replaced), experiential equivalence (as daily experiences may differ in different cultures, items may have to be replaced by something addressing a similar intent in the target culture), and conceptual equivalence (checking that the ‘same’ word does not hold a different conceptual meaning in the culture) (Beaton et al. 2007). (5) The final step is to pre-test the new instrument with a small sample reflecting the target population. Beaton et al. recommended using 30–40 persons. Variants of these sequences exist, e.g. Borsa et al. described the expert review as taking place before the back-translation and also suggested an instrument evaluation by the target population (Borsa, Damásio, and Bandeira 2012). Borsa et al. also recommended an additional stage that is normally not included in the adaptation process to confirm whether the instrument structure is stable compared to the original instrument, namely evaluation of the factorial structure of the instrument accomplished by exploratory and confirmatory factor analyses. Recent research has suggested that exploratory structural equation modelling (ESEM) combines the flexibility of exploratory factor analysis with access to the typical parameters and statistical advances of confirmatory factor analysis/structural equation modelling (Morin, Marsh, and Nagengast 2013). While confirmatory factor analysis has largely superseded exploratory factor analysis, ESEM has been argued to be a more general framework and less restricted than confirmatory factor analysis and structural equation modelling and, therefore, preferable in clinical psychology research (Marsh et al. 2014).

Knowledge gap identified in instruments for assessing generic skills

To sum up the knowledge gap, there is a lack of validated instruments in English for assessing the learning of generic skills, especially instruments which cover the collaborative aspects of modern knowledge work practised in authentic course settings. Poorly adapted instruments may risk generating inconsistent and unreliable data (Borsa, Damásio, and Bandeira 2012). The trustworthiness of research collected in one language and presented in another relies heavily on translation-related decisions (Chen and Boore 2010). Using established measures allows for comparison of findings in different countries, but usually little is reported about the translation and adaptation process of the instruments used, making it difficult to evaluate the equivalency and quality of the instruments and the data that they have generated (van Widenfelt et al. 2005). The Collaborative Knowledge Practices Questionnaire (CKP) has been demonstrated to be a valid tool in Finnish but a validated, English version is lacking.

Aim

This study aimed to cross-culturally translate and adapt the original instrument into English and perform an exploratory structural equation modelling (ESEM) analysis in order to investigate whether the factorial solution of the original instrument also works in English in higher education courses in international settings.

The cross-cultural translation and adaptation of the instrument examined the methodological questions about the steps, criteria, and challenges of the translation and adaptation of the questionnaire to another context while the analysis of the factorial solution aimed to answer whether the same factorial solution could be found for the data in the new context.

Methods

Cross-cultural translation and adaptation of the original instrument

The original instrument was translated into English in line with the guidelines proposed for cross-cultural adaptation of self-report measures (Beaton et al. 2000; Borsa, Damásio, and Bandeira 2012). First, two translations of the instrument were created independently (see Figure 1). One of the forward translations (T1) was done by two bilingual translators in collaboration while the other forward translation (T2) was done independently by a third bilingual translator. All three translators were knowledgeable of the topic. To synthesise the results, a fourth bilingual person was added to the team who acted as a mediator in discussions of the differences in the translations. Issues were resolved by consensus and one common translation (T-12) was produced. Working from the T-12 version of the questionnaire, and blind to the original version, two new bilingual translators translated back from English into Finnish to ensure validity by enabling comparison of the new translation to the original version. The back-translations (BT1 and BT2) were produced by two bilingual persons with Finnish as their mother tongue. The two translators worked independently of each other and were neither aware nor informed of the concepts explored to avoid information bias and to elicit unexpected meanings of the items in the translated questionnaire (T-12) (Beaton et al. 2007; Guillemin, Bombardier, and Beaton 1993). All different versions of the instruments were collected in a document and the different versions were colour-coded and organised in one document for easier comparison. An expert committee consisting of the translators, the mediator as well as an additional person who is one of the original developers reviewed the material. The importance of involving the original instrument’s authors in the evaluation has been emphasised by Borsa and colleagues (Borsa, Damásio, and Bandeira 2012). The differences between the different versions were discussed until reaching a consensus on a pre-final version.

Figure 1. The cross-cultural translation and adaptation process.

Altogether 57 students taking part in three media engineering courses in Finland participated in pretesting of the instrument. The instrument worked in a satisfactory manner; no complaints about the comprehensibility of items were expressed and all respondents responded to each item; therefore, the pilot did not lead to any further modifications of the instrument.

Participants

In the evaluation of the factorial solution, data were collected from two research-intensive universities and one applied sciences university. Students (n = 169) in seven courses responded to the CKP questionnaire at the end of their courses. Four courses were organised in Sweden (medical education and online tools for teaching, respectively) and three courses in Finland (two problem solving courses in the educational field and one in engineering) for international students in English. In all, 59 participants were male, 108 female (63.9%) and 2 responded as other (1.2%). Age varied from 19 to 61 years (M = 39.3, SD 11.07, Median = 40). The response rate was 37.3%.

The sample included courses with intensive virtual or face-to-face collaboration with participants from diverse international backgrounds.

Data collection

All documents from the translators, observations made at review meetings, different versions of the questionnaire items and suggestions for changes and comments were stored using a web-based software with a revision history functionality.

The CKP questionnaire (Muukkonen et al. 2019, 2017) was used for data collection. Students evaluated how 27 statements corresponded to their learning during the course. ‘During the course I have learned … ’ e.g. ‘to develop ideas further together with others’. The statements were assessed using a five-point Likert-scale ranging from 1 (not at all) to 5 (very much). The statements make up the seven scales of the questionnaire: (1) Learning to collaborate on shared objects, (2) Integrating individual and collaborative working, (3) Development through feedback, (4) Persistent development of knowledge objects, (5) Understanding various disciplines and practices, (6) Interdisciplinary collaboration and communication and (7) Learning to exploit technology.

Responses to the questionnaires were collected at the end of each course using web-based surveys. Participants who gave informed consent were included in the study.

Data analysis

The data were screened for outliers and unmotivated responses which were removed, resulting in N = 163 included in the analysis. We conducted an exploratory structural equation modelling (ESEM) (Asparouhov and Muthén 2009; Marsh et al. 2014, 2009) factor analysis with the CKP items as ordered categorical variables. When a structural model includes more than a few cross-loadings recognised by theory or by item content, ESEM can be preferred to conventional confirmatory factor analysis (CFA) which usually constrains items to load only on one factor and can lead inflated factor correlations in applied research (Marsh et al. 2014, 2009). Since none of the respondents had selected the alternative ‘Not at all’ as responses to two of the items (10 and 12), these only had four ordered categories while the other 25 items had five categories. In the rotation, target loadings of one was designated to items belonging to the same hypothesised factor and target loadings of zero to other items (see Table 1).

Table 1. ESEM solution for a target rotated 7 factor model: standardised loadings, factor variances, and correlations.

	Factor loadings
Item	1	2	3	4	5	6	7
During the course I have learned …
L08 to coordinate the development of products (e.g. plans, reports, models) together with others	.453 T	.129	.044	.259	.009	.066	.166
L10 to take responsibility for the shared group work	.541 T	.383	−.107	−.012	−.073	−.009	.111
L24 to plan the collaborative work	.597 T	.202	.200	−.196	−.104	.381	.058
L26 to develop ideas further together with others	.206 T	.164	.516	−.045	.086	.008	.166
L06 to understand how important the expertise of others is when developing products	.003	.314 T	.053	.181	.349	.119	.047
L11 to define subgoals for the collaborative work	.368	.844 T	.032	.090	−.305	.047	−.332
L12 to understand the benefits of working in collaboration	.133	.679 T	.010	−.007	−.027	−.077	.246
L20 to accomplish challenging tasks in collaboration with others	.349	.453 T	.190	−.052	−.030	.061	.107
L19 to receive feedback on my products (e.g. plans, reports, models) for developing them further	−.016	.218	.505 T	.252	.062	.144	−.277
L23 to comment on the work of others	.240	.055	.700 T	−.254	−.135	.250	−.057
L27 to understand the value of commenting work in progress	.017	.210	.744 T	.079	−.030	−.333	.118
L29 to finalise products with patience	.386	−.202	.474 T	.280	.222	−.096	−.124
L01 to evaluate the development of a shared product	−.176	.108	.011	.849T	−.104	.203	.012
L02 to work on products that are used later by others/myself	−.235	.186	.059	.715 T	−.127	.156	.103
L09 to work on the shared products by improving them iteratively	.379	−.215	.146	.510 T	−.113	−.037	.210
L16 to evaluate how much effort is needed to develop a product	−.045	.089	.110	.390 T	.275	−.199	.137
L03 new aspects about the practices of different organisations	−.037	−.139	−.129	.296	.430 T	.436	.138
L15 the practices of people with different kinds of expertise	.075	.352	−.230	−.161	.875 T	.369	.042
L25 about the practices of work-life experts	−.066	−.236	.428	.121	.579 T	.453	−.017
L28 how useful it is to learn the working practices of other fields and organisations	.151	.173	.203	−.071	.626 T	.264	−.076
L04 to ask questions relating to the practices of another field	−.006	.235	−.079	.230	.486	.409 T	−.079
L13 to present my expertise to representatives of another field	.224	.096	.138	.096	.439	.307 T	−.097
L18 to collaborate with representatives of other fields	.199	−.011	.088	−.102	.582	.487 T	.185
L05 to use technology to advance collaborative work	.200	.084	−.360	−.008	−.023	.172	.882 T
L07 to use various digital applications and use them together whenever needed	.026	−.071	−.049	.053	−.093	.014	.935 T
L14 to understand the possibilities of digital technology better than before	.003	.072	−.018	.077	.073	−.130	.765 T
L17 to develop products collaboratively by using technology	.220	−.044	.071	.355	−.053	−.108	.507 T
	Factor correlations (upper diagonal) and their p-values (lower diagonal, italics)
	F1	F2	F3	F4	F5	F6	F7
F1 Learning to collaborate on shared objects		.441	.390	.585	.582	−.068	.493
F2 Integrating individual and collaborative working	.071		.676	.557	.395	.362	.534
F3 Development through feedback	.046	< .001		.425	.365	.378	.427
F4 Persistent development of knowledge-objects	< .001	< .001	< .001		.511	.181	.401
F5 Understanding various disciplines and practices	< .001	.011	.002	< .001		−.258	.399
F6 Interdisciplinary collaboration and communication	.680	.041	.001	.057	< .001		.116
F7 Learning to exploit technology	< .001	< .001	< .001	< .001	< .001	.291

Absolute factor loadings ≥ .30 are bold. Subscript _T indicates target loading of 1.

Results

In certain cases, the different translators consistently selected differing terms in the translations. For instance, the Finnish word ‘arvioida’ was translated as either ‘assess’, ‘evaluate’ and ‘consider’ and, depending on the part of the instrument, decisions needed to be made about which translation corresponded best to the original, intended meaning (e.g. ‘to evaluate the development of a shared product’ was selected), or which translation worked best stylistically in English (‘to understand how important the expertise of others is … ’). Another observation was that decisions needed to be made whether to follow the original instrument strictly or strive for consistency in the new translation. For instance, during the translation process, discussions were conducted about the importance of keeping (seemingly unnecessary) variations of the ‘same’ concept in the original instruments such as mixing singular and plural forms of a concept in different items. Similar decisions needed to be made in several other cases of observed differences in the translations regarding closely related concepts such as expertise/competency/skills; develop/produce/construct/prepare; field/domain; understanding/seeing/perceiving; knowledge/information/what I have learned, etc.

Certain concepts in the original CKP were particularly challenging to translate because a corresponding word did not exist or because a translation would place emphasis on an aspect which was not intended in the original language. One example of this was one of the most central phrases, namely ‘kehittää tuotoksia’ in the original instrument. The phrase is roughly equivalent to ‘developing products’ or ‘developing outputs’. Numerous translations needed to be considered. For instance, the word ‘product’ was a strong candidate but was considered to sound slightly more businesslike in English and not as being obvious in an educational context as the original phrase. ‘Product’ as well as ‘object’ or ‘thing’ also imply something more concrete than the original phrase does and thereby risked excluding more abstract results such as plans and reports. Alternative terms that were considered were the more general ‘artifact’, ‘knowledge object’ and ‘object’, but these were considered to be more theory-laden and risked appearing unfamiliar to respondents. Another possible translation was ‘assignment’ but this word was perceived as sounding too school-related with too little focus on the concrete outcomes that the original included. Yet another candidate word was ‘production’ but it was also considered abstract and risked unintendedly directing respondents’ thinking to the outputs of specific businesses such as film or television. It was finally decided that the best translation was to use ‘product’ in combination with examples to help respondents understand the intended meaning: ‘the development of products (e.g. plans, reports, models)’. The use of such examples was also included in the original version of the instrument.

The results of the ESEM solution for the target rotated seven factor model are presented in Table 1. The fit of the ESEM model was good (root mean square error of approximation [RMSEA] = .047, comparative fit index [CFI] = .989, Tucker–Lewis index [TLI] = .979). Threshold levels RMSEA < .07, CFI > .95, and TLI > .95 are considered indicative of good fit (Hooper, Coughlan, and Mullen 2008). All items except one (item 26, which nevertheless had a loading of .206), loaded on their respective factors although many cross-loadings were also present (Table 1). The factors correlated somewhat strongly, but this was expected based on the original factorial solution.

Discussion

Translation of an instrument calls to attention the fact that different languages cover phenomena in differing ways. While striving for equivalence to the original instrument and at the same time attempting to achieve comprehensibility and consistency in the English version, the translation work needed to balance goals that were occasionally in conflict. Firstly, the main goal was naturally to follow the phrasing of the original instrument. At the same time, literal translation is not always ideal as such translation may result in a lack of language fluency in the target language and therefore a literal translation needed to be replaced with a more appropriate and comprehensible wording in certain items. A second issue concerned the opposite case, when observations were made that the original instrument was not entirely consistent in its phrasing or form. Should the new instrument keep seemingly unintentional variation of wordings or should it strive for being consistent in its use of a term? A consistent use of wording may in itself be helpful for future respondents but such consistency would mean taking a step away from following the original version. The third issue concerned the need to balance the tension between creating an accessible instrument without jargon versus retaining the precision of the theoretical concepts underlying the original item. This posed a challenge which was different to the cases usually discussed in the literature about cross-cultural translation and adaptation as it was not related to difficulties related to idiomatic, linguistic, contextual or otherwise cultural adaptation but, instead, having to do with the fact that the phenomenon referred to by the questionnaire item was novel. The difficulty was thus not a difficulty of identifying the appropriate everyday language expression in English but rather the fact that the concept itself is innovative in educational context in a way which diverges from everyday language. Conceptualising ‘learning’ as a collaborative process involving the creation of shared objects over time, as the Trialogical approach to learning (Paavola and Hakkarainen 2005) does, is atypical and not quite the ordinary, everyday understanding of what ‘learning’ refers to. As the approach reconceptualises or expands the concept of learning, individual questionnaire items probing about learning are difficult to phrase in ‘ordinary language’ if they are to keep the intended meaning. Therefore, certain such terms may have to, at least to a certain degree, initially be experienced as unfamiliar by the target group. This challenge was handled by accompanying such questionnaire items with explanations and examples. Such modifications were also done in the original instrument, thereby being a case of ‘decentering’, i.e. opening the original instrument for improvement during the translation procedure (van Widenfelt et al. 2005).

As mentioned in the background, cultural and contextual aspects may also need to be considered when adapting an instrument into a new context (Borsa, Damásio, and Bandeira 2012). The organisation of courses and the design of learning activities may vary significantly between different educational contexts. Rules and legislation as well as expectations and attitudes towards students, learning and teaching may differ considerably. Innovation and creativity among learners may be more or less encouraged by teachers and embraced by the students themselves. Attitudes towards the use of technologies may differ. And examinations may be strictly individual or allow different degrees of collaboration among students. Such attitudes and expectations may affect students’ interactions and therefore the possibilities of implementing the learning activities addressed by the CKP questionnaire. It is therefore conceivable that very competitive university cultures may undermine the development of fruitful collaborative interaction. Moreover, if students have little agency and initiative in orchestrating their studying and learning practices, the setting may not be appropriate for the activities indicated by the items of the questionnaire.

Recent literature on the adaptation of instruments stresses adding an evaluation of the instrument’s factorial structure to the adaptation process (Borsa, Damásio, and Bandeira 2012); consequently, an ESEM analysis was added to the process of developing an international version of the CKP. A test of the factorial solution of CKP provided evidence that the factorial solution found in the Finnish language sample could be applied to the international English language sample, as the ESEM solution model provided a good fit for the data. We found that items designated to factors 5 (Understanding various disciplines and practices) and 6 (Interdisciplinary collaboration and communication) in the present data loaded more onto both factors than in the original data. In the original data, students came more often from one discipline, i.e. media engineering and life sciences, even though there were cases where teachers were from different disciplines or external customers provided experiences from their respective organisational practices. A true case of interdisciplinary collaboration would entail that students worked in interdisciplinary teams during the course. We expect that the reason for the difference in loadings lies in that most of the courses in the present data involved interdisciplinary collaboration among the students and, therefore, the participants did not differentiate between the two factors regarding understanding various disciplines and interdisciplinary collaboration. To further examine the differences between these two factors, additional data is needed representing both types of pedagogical designs.

Conclusion

The CKP questionnaire aims to assess new aspects of a generic competence related to collaboration, interdisciplinarity and technology use. In the literature, competencies are mostly defined and measured as pertaining to solely individual skills or abilities. Here, the aim is to advance understanding of collective aspects of competence in knowledge work, how these develop in various pedagogical settings and cultural contexts. The questionnaire provides an instrument to make comparisons across local and intercultural contexts. The questionnaire adaptation process described the conceptual challenges related to adjusting the theoretically grounded questionnaire items to another language. However, this also aided to clarify the intended meaning of the items in both languages and cultural contexts. The results of the statistical validation provided evidence that the latent factor model found in the original instrument provided a good fit also for the adapted questionnaire. Next, it would be valuable to empirically test the questionnaire in diverse contexts.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was partly supported by the Academy of Finland [Grant 285 806] and the European Commission (KNORK 402765 543154-LLP-1-2013-1-FI-KA3-KA).

Notes on contributors

Klas Karlgren

Klas Karlgren is a senior researcher in the areas of simulation and technology-enhanced learning. He leads the MINT research team at the department of Learning, Informatics, Management and Ethics at Karolinska Institutet, a medical university in Stockholm, Sweden. He is also a researcher and research coordinator at the Södersjukhuset hospital in Stockholm, Sweden and associate professor at the SimArena simulation center at the Western Norway University of Applied Sciences, Ber

Minna Lakkala

Minna Lakkala Ph.D. (psychology), works as a university researcher at the University of Helsinki. She is specialized in technology-enhanced teaching and learning, knowledge creation pedagogy and competence learning in all educational levels. She has participated in several national and international research and development projects concerning educational practices since 1998.

Auli Toom

Auli Toom PhD, is a professor of higher education and director of the Centre for University Teaching and Learning (HYPE) at the University of Helsinki, Finland. She is a visiting professor at the University of Tartu, Estonia. Dr. Toom is also the director of the doctoral programme PsyCo (Psychology, Learning and Communication). Dr. Toom is the vice president of the Finnish Educational Research Association. Her research interests include teacher knowing, competence, expertise and agency among students and teachers. She investigates them in basic education, teacher education and higher education contexts. She leads several research projects on higher education and teacher education.

Liisa Ilomäki

Liisa Ilomäki EdD., docent (adjunct professor) in education and technology, works at the University of Helsinki, Finland. Her research interests include the use of digital technologies in learning and teaching and the development of students’ knowledge work competence. Ilomäki has been involved in several European and national research projects in technology-enhanced learning.

Pekka Lahti-Nuuttila

Pekka Lahti-Nuuttila is a product manager at the Hogrefe Psykologien Kustannus Oy. Currently he also does research on developmental language disorder in the Helsinki Longitudinal SLI Study at the department of Otorhinolaryngology and Phoniatrics, Head and Neck Surgery at the Helsinki University Hospital and the University of Helsinki. The research is part of his doctoral studies at the department of Psychology and Logopedics in the Faculty of Medicine, University of Helsinki

Hanni Muukkonen

Hanni Muukkonen is a professor in educational psychology at the Faculty of Education, University of Oulu, Finland. Her current research focuses on collaborative learning, knowledge creation, learning analytics, and technology in education.