Using Generalizability theory to assess the score reliability of the Special Ability Selection Examinations for music education programmes in higher education

Abstract

The ‘Special Ability Selection Examination’ (SASE), which is used to select appropriate students for the music education departments of educational faculties in Turkey, has many subsections and must evaluate highly competitive cohorts of students according to a broad range of criteria. The test consists of three subsections, with a large number of tasks in each subsection. As a result, many factors can affect the success of the candidate, including the number of raters evaluating the test. In such tests, Generalizability theory, or G theory, can be used to reliably analyse these multiple facets such as the tasks, raters, etc. This article uses G theory to analyse one version of the SASE that was assessed by four raters and given to 249 candidates of the music education department at İnönü University for the 2003–2004 academic year. G theory’s multivariate c^• × t° × r^• pattern is the most appropriate statistical methodology in this case, since the test consisted of subsections and involved multiple raters for which interaction effects should be investigated. In the first portion of this study, the reliability of the SASE is analysed using G theory. In the second, the Decision study methodology is used to examine alternative ways of altering the length of the test and reducing the number of raters so that cost and time expenditure can be minimized in future issuances of the test.

Keywords:

• generalizability theory
• many facet measurement
• decision study
• reliability
• inter rater reliability
• special ability selection
• examinations