Designing and Scaling Level-Specific Writing Tasks in Alignment With the CEFR: A Test-Centered Approach

Figures & data

TABLE 1 Model of Communicative Competencies in the NES

Communicative Skills	Availability of Linguistic Resources
Functional Communicative Competencies	Vocabulary
Reading comprehension	Grammar
Listening and aural-visual comprehension	Pronunciation and Intonation
Speaking	Orthography / Spelling
(a) Participation in conversations
(b) Connected speech
Writing
Mediation
Intercultural Competencies
Sociocultural orientation knowledge
Sensitivity for cultural diversity
Practical skills for intercultural encounters
Methodological Competencies
Text reception (reading comprehension and listening comprehension)
Interaction
Text production (speaking and writing)
Learning strategies
Presentation skills and skills for media usage
Learning awareness and organization of learning processes
Note. NES = National Educational Standards, internal unpublished translation.

TABLE 2 Overview Over the National Educational Standards Writing Competency Descriptors for the Mittlerer Schulabschluss

General Descriptors	Specific Descriptors
The learners can write coherent texts on familiar topics of interest (B1).	The learners can …
	1. take notes if somebody asks for information or explains a problem (B1+),
	2. convey messages, simple information and thoughts in personal letters (B1),
	3. phrase simple targeted standardised letters and e-mails, for example enquiries, applications (B1),
	4. write uncomplicated, detailed texts on a number of different topics on their areas of interest, for example reports, stories, descriptions (B1),
	5. write short simple essays on topics of general interest (B1),
	6. write short reports on familiar topics, use them to pass on information, provide reasons for action and comment (B 1+).
Note. National Educational Standards, internal unpublished translation.

TABLE 3 Test Specifications

CEFR Level	Task Characteristics	Specific Purpose and Text Types	Range of Discourse Types Demanded	Output: Textual / Linguistic Expectations
A1	• very short prompts on concrete topics	Specific Purpose:• ask for or pass on personal details	• narrative• descriptive• phatic• simple expository	• very short and simple output• groups of words• simple isolated phrases• short sentences• highly frequent, elementary vocabulary
	• concrete, very simple and familiar topics
	• all expected content points clearly stated	Text Types:• write a list• fill in forms• write a postcard
B1	• short prompts on concrete or slightly abstract topics	Specific Purpose• pass on information• provide reasons for actions and comments	• narrative• descriptive• instructive	• short, straightforward, detailed texts (110–140 words)• logical development of ideas• coherent text
	• topics of personal as well as general interest	Text Types:• write notes and messages	• expository• simple  argumentative	• common, frequent grammatical features
	• expected content points listed	• write personal letters, simple formal letters and emails		• task-specific vocabulary
		• write reports/ articles
C1	• longer, also complex prompts	Specific Purpose	• narrative	• longer, clear, detailed, well-structured texts (300–350 words)
	• concrete as well as abstract, also unfamiliar topics	• cover all sorts of purposes	• descriptive	• effective structuring of text: underlining relevant issues, rounding off with an appropriate conclusion
	• expected content points not listed separately	• write about complex subjects	• instructive	• style appropriate to the reader in mind
		• expand and support points of view at some length• give reasons and relevant examples	• expository• argumentative• phatic	• complex grammatical structures
		Text Types		• idiomatic language use, precise vocabulary
		• write texts of all types, formal and informal		• emotional, allusive and joking language usage, if demanded by task
Note. CEFR = Common European Framework of Reference.

TABLE 4 Percentage of Student Responses Classified as “Pass” by Sample, Task, and Rater

	Task ID	CEFR Classi-fication	Rater ID
Sample			1	2	3	4	5	6	7	8	9	10	11	12	13	Task M
	4	A1	.70	.80	.71	.74	.78	.67	.52	.64	.86	.57	.72	.86	.89	.73
	7	A1	.61	.71	.60	.84	.63	.83	.70	.85	.81	.42	.67	1.00	.70	.72
	9	A1	.43	.44	.52	.74	.64	.57	.46	.55	.79	.63	.72	.69	.78	.61
	14	A2	.62	.47	.39	.58	.56	.35	.17	.27	.30	.32	.44	.40	.52	.41
HSA	8	A2	.11	.39	.42	.67	.32	.13	.24	.44	.52	.41	.43	.39	.28	.37
	1	A2	.22	.39	.25	.27	.50	.25	.25	.33	.00	.29	.38	.29	.40	.29
	17	B1	.00	.00	.09	.33	.24	.00	.00	.05	.00	.60	.00	.00	.13	.11
	13	B1	.13	.00	.10	.08	.08	.00	.05	.00	.06	.00	.07	.00	.12	.05
	19	B1	.06	.09	.08	.06	.08	.00	.00	.08	.00	.00	.13	.00	.11	.05
Rater M			.32	.37	.35	.48	.43	.31	.27	.36	.37	.36	.40	.40	.44	.37
	7	A1	.97	1.00	.98	1.00	1.00	1.00	.98	1.00	.99	.93	1.00	1.00	.99	.99
	1	A2	.91	.87	.91	.80	.84	.84	.96	.96	.93	.87	1.00	.88	.96	.90
	2	A2	.61	.91	.86	.93	.73	.90	.75	.92	.85	.82	.94	.88	.95	.85
	3	A2	.78	.90	.82	.85	.76	.86	.72	.87	.87	.79	.88	.74	.96	.83
	12	A1	.72	.87	.72	.90	.80	.77	.82	.83	.83	.88	.85	.76	.97	.82
	10	A2	.78	.86	.79	.89	.88	.82	.70	.87	.88	.79	.85	.80	.62	.81
MSA	17	B1	.51	.73	.73	.80	.82	.63	.68	.93	.63	.66	.75	.50	.45	.68
	16	B1	.56	.68	.46	.59	.66	.68	.68	.56	.48	.56	.53	.64	.69	.60
	15	B1	.51	.71	.64	.35	.69	.55	.50	.47	.77	.68	.52	.46	.73	.58
	5	B2	.31	.15	.36	.18	.47	.13	.24	.14	.14	.18	.32	.19	.15	.23
	6	B2	.38	.08	.26	.07	.34	.06	.20	.04	.08	.19	.24	.16	.18	.18
	18	C1	.13	.10	.14	.03	.23	.06	.11	.02	.04	.09	.18	.06	.11	.10
	11	C1	.15	.09	.18	.00	.01	.01	.06	.01	.07	.11	.12	.06	.07	.07
Rater M			.81	.88	.60	.57	.63	.56	.57	.59	.58	.58	.63	.54	.60	.63
Note. CEFR = Common European Framework of Reference; HAS = Hauptschulabschluss; MSA = Mittlerer Schulabschluss.

TABLE 5 Aggregated Variance Components Estimates for HSA and MSA Samples

	HSA Sample			MSA Sample
	VC	%	SD	VC	%	SD
Student	0.0490	24.3	0.0247	0.0225	8.4	0.0136
Task	0.0182	9.0	0.0194	0.0996	37.3	0.0609
Criterion	0.0007	0.4	0.0014	0.0009	0.4	0.0021
Rater	0.0024	1.2	0.0028	0.0058	2.2	0.0055
Student × Task	0.0347	17.2	0.0242	0.0313	11.7	0.0139
Student × Criterion	0.0000	0.0	0.0009	0.0016	0.6	0.0018
Student × Rater	0.0021	1.1	0.0019	0.0020	0.7	0.0023
Task × Criterion	0.0021	1.0	0.0020	0.0043	1.6	0.0038
Task × Rater	0.0037	1.8	0.0026	0.0118	4.4	0.0099
Criterion × Rater	0.0007	0.3	0.0012	0.0008	0.3	0.0008
Student × Task × Criterion	0.0155	7.7	0.0052	0.0122	4.6	0.0069
Student × Task × Rater	0.0173	8.6	0.0053	0.0190	7.1	0.0052
Student × Criterion × Rater	0.0004	0.2	0.0013	0.0011	0.4	0.0020
Task × Criterion × Rater	0.0043	2.1	0.0025	0.0037	1.4	0.0032
Error	0.0507	25.2	0.0151	0.0507	19.0	0.0115
No. of booklets		6			9
No. of ratings		8,741			13,085
Note. HAS = Hauptschulabschluss; MSA = Mittlerer Schulabschluss; VC = aggregated variance components estimates.

FIGURE 1 Wright-map for multifaceted Rasch analysis of Mittlerer Schulabschluss sample data (from Rupp & Porsch, 2010, p. 60. Reprinted with permission).

TABLE A1 Illustrative CEFR Writing Descriptors for Level B1

Descriptors	Scale	CEFR Reference
Can write straightforward connected texts on a range of familiar subjects within his field of interest, by linking a series of shorter discrete elements into a linear sequence.	Overall Written Production	p. 61
Can convey information and ideas on abstract as well as concrete topics, check information and ask about or explain problems with reasonable precision. Can write personal letters and notes asking for or conveying simple information of immediate relevance, getting across the point he/she feels to be important.	Overall Written Interaction	p. 81
Has a sufficient vocabulary to express him/herself with some circumlocutions on most topics pertinent to his/her everyday life such as family, hobbies and interests, work, travel, and current events.	Vocabulary Range	p. 112
Communicates with reasonable accuracy in familiar contexts; generally good control though with noticeable mother tongue influence. Errors occur, but it is clear what he/she is trying to express.Uses reasonably accurately a repertoire of frequently used “routines” and patterns associated with more predictable situations.	Grammatical Accuracy	p. 114
Note. CEFR = Common European Framework of Reference.

TABLE B1 Development of Test Specifications, Illustrated for Specific Purpose, Level B1

Draft 1	Draft 2	Final
Specific Purpose: To determine whether the test taker is able to:	Specific Purpose: To determine whether the test taker is able to:	Specific Purpose: To determine whether the test taker is able to:
- write straightforward connected and detailed texts (NES 13f, CEFR 61)	1. write notes and messages	1. Write straightforward detailed texts (connected and in linear sequence) on topics of personal as well as general interest (stories, descriptions including experience, feelings etc.)
- take messages (NES 14, CEFR 84)	2. write personal letters, simple formal letters and emails conveying messages, simple information and thoughts	2. Write notes and messages
- write notes (NES 14, CEFR 96)	3. write straightforward detailed texts (connected and in linear sequence) on topics of personal as well as general interest (reports, stories, descriptions including experience, feelings etc.)	3. Write personal letters, simple formal letters and emails conveying messages, simple information and thoughts
- convey messages, simple information and thoughts (NES 14, CEFR 84)	4. write short reports on familiar topics passing on information, providing reasons for actions and comments (B1+)	4. Write short reports/articles on familiar topics passing on information, providing reasons for actions and comments (B1+)
- phrase simple targeted standardised letters and emails (NES 14, CEFR 83)
- summarize, report and give his/her opinion (CEFR 62)
Note. NES = National Educational Standards; CEFR = Common European Framework of Reference.

TABLE C1 Rating Instrument for Level B1

Level	Task Fulfilment	Organisation (Structure and Cohesion Should Have Equal Weighting)	Grammar (Accuracy Should Be Treated in Relation to Range)	Vocabulary (Accuracy Should Be Treated in Relation to Range)
B1		Structure / Thematic development	Range	Range
	1. Most of the expected content points are mentioned, some of them in more depth or some content points are dealt with in more depth while others might be missing / might be just shortly addressed.	1. Produces a straightforward connected piece of text (narrative or description) in a reasonably fluent manner, or links a series of shorter discrete simple elements into a linear sequence of points in a reasonably fluent manner.	1. Uses a range of frequently used structures (such as tenses, simple passives, modals, comparisons, complementation, adverbials, quantifiers, numerals, adverbs).	1. Shows a sufficient range of vocabulary (beyond basic) to express him/herself in familiar situations; some circumlocutions may occur in unfamiliar situations.
		Thematic development shows a logical order and is rounded off.
		Longer texts might compensate for some jumpiness or a missing ending.
		Organisation in paragraphs not required, but might compensate for flaws in thematic development.
		Language / Cohesion		Accuracy
	2. Most of the ideas are relevant to the task.	2. Uses a number of common cohesive devices throughout the text, such as articles, pronouns, semantic fields, connectors, discourse markers [like “so” (consecutive), “in my opinion”].	2. Sentence patterns show simple variations (e.g., subordinate and coordinate clauses often beginning with “when,” “but”; relative clauses and if-clauses).	2. Shows good control (i.e., adequate and appropriate use) of basic vocabulary.
				Code 3 (“shows control of more than basic vocab”) cannot be awarded if V1 is allotted code 1.
			Accuracy
	3. Register and tone are consistent and appropriate for the target audience.	3. Shows reasonable control of common cohesive devices.	3. Structures and sentence patterns shown in the script are used reasonably accurately.	3. Some non-impeding^b errors occur.
		The use of more elaborate cohesive devices may sometimes impede communication.
	4. Meets text type requirements (i.e. following a standard format).		4. Some local errors^b occur, but it is clear what he/she is trying to express.	4. Impeding^c errors may occur occasionally, especially when expressing more complex thoughts or handling unfamiliar topics and situations.
				Errors may occur in the field of collocations and complementation.
				Some spelling errors may occur.
				Occasionally mother tongue influence may be noticeable.
	5. Communicative effect is mainly achieved, i.e., the message is conveyed although occasional difficulty may be experienced by the reader.		5. Few global errors may occur, especially when using more complex structures / sentence patterns (e.g., relative clauses, if-clauses, passives and indirect speech).
			Occasionally mother tongue influence may be noticeable.
Note. The statements in italics are rating guidelines and are not actually rated. Documents consulted: Manual for linking examination to the CEFR (Council of Europe, 2003, p. 82); Into Europe Rating Scale (Tankó, 2005); CEFR (Council of Europe, 2001); DESI Rating Scales (Harsch, 2007). Notes on rating procedure: Each numbered descriptor is rated by the following approach: Student text shows less than the demanded features (below): Code 1. Student text shows the demanded features (pass): Code 2.
^a Local errors are grammatical errors within one sentence that do not hinder understanding (e.g., mixing up of tenses, forgetting to mark agreement, problems with subordinate clauses, errors in word order). It is usually clear what the writer wants to express.
^b Nonimpeding errors are those lexical / spelling errors that can be resolved spontaneously.
^c Impeding errors are those lexical / spelling errors that are irresolvable or take a great deal of effort to resolve.

Rupp , A. A. and Porsch , R. 2010 . “ Standard-setting item pool ” . In Calibrating standards-based assessment tasks for English as a First Foreign Language—Standard-setting procedures in Germany , Edited by: Harsch , C. , Pant , H. A. and Köller , O. 39 – 60 . Münster, , Germany : Waxmann .

 Google Scholar

Council of Europe. (2003). Relating language examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEF). Manual, preliminary pilot version. Strasbourg, France: Author. http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/Manual.html (http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/Manual.html)

 Google Scholar

Tankó , G. 2005 . Into Europe: Prepare for modern English exams. The writing handbook , Budapest, , Hungary : Teleki László Foundation .

 Google Scholar

Council of Europe. (2001). Common European reference framework for languages. Strasbourg, France: Author. http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/common_framework.html (http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/common_framework.html)

 Google Scholar

Harsch , C. 2007 . Der gemeinsame europäische Referenzrahmen für Sprachen: Leistung und Grenzen [The Common European Framework of Reference for Languages: Strengths and limitations] , Saarbrücken, , Germany : VDM .

 Google Scholar