Scoring Behavior of Native vs. Non-native Speaker Raters of Writing Exams

Abstract

In performance testing, where judges provide scores on examinees' abilities, program administrators may seek to ensure that raters are consistent in their interpretations of the scoring criteria. A Rasch analysis not only allows us to examine differences in rater consistency, but also to check for interactions between rater characteristics and scoring behavior. In this paper, we present the results of an analysis of rater severity in assessing students' writing ability. Our main focus was to examine differences between native speaker (NS) and non-native speaker (NNS) raters. Results showed that raters differed in terms of severity, with NNS raters as a group more severe than NS raters. In addition, the severity of NNS raters varied more than that of the NS raters. Bias analysis indicated some rater-examinee bias and rater-domain bias in both NS raters and NNS raters, with the majority found among the NNS raters. The paper provides implications for training NNS raters. (153 words)