Publications
Detailed Information
Effects of rating criteria order on the halo effect in L2 writing assessment: a many-facet Rasch measurement analysis
Cited 4 time in
Web of Science
Cited 2 time in Scopus
- Authors
- Issue Date
- 2020-11-10
- Publisher
- Springer Open
- Citation
- Language Testing in Asia. 2020 Nov 10;10(1):16
- Keywords
- Halo effect ; Rating criteria order ; Rating scale model (RSM) ; Many-facet ; Rasch measurement (MFRM)
- Abstract
- The halo effect is raters undesirable tendency to assign more similar ratings across rating criteria than they should. The impacts of the halo effect on ratings have been studied in rater-mediated L2 writing assessment. Little is known, however, about the extent to which rating criteria order in analytic rating scales is associated with the magnitude of the group- and individual-level halo effects. Thus, this study attempts to examine the extent to which the magnitude of the halo effect is associated with rating criteria order in analytic rating scales. To select essays untainted by the effects of rating criteria order, a balanced Latin square design was implemented along with the employment of four expert raters. Next, 11 trained novice Korean raters rated the 30 screened essays with respect to the four rating criteria in three different rating orders: standard-, reverse-, and random-order. A three-facet rating scale model (L2 writer ability, rater severity, criterion difficulty) was fitted to estimate the group- and individual-level halo effects. The overall results of this study showed that the similar magnitude of the group-level halo effect was detected in the standard- and reverse-order rating rubrics while the random presentation of rating criteria decreased the group-level halo effect. A theoretical implication of the study is the necessity of considering rating criteria order as a source of construct-irrelevant easiness or difficulty when developing analytic rating scales.
- ISSN
- 2229-0443
- Language
- English
- Files in This Item:
Item View & Download Count
Items in S-Space are protected by copyright, with all rights reserved, unless otherwise indicated.