The Study on the Rater Reliability of Three Scoring Methods in Assessing Argumentative Essays: Holistic, Analytic, and Multiple-Trait Scoring Methods

Abstract

Various studies have been conducted to minimize the subjectivity and increase the accuracy in assessing written texts, and the present study focused on the scoring rubrics which were the basic criteria for evaluating writing. Three different scoring rubrics (holistic, analytic and multiple-trait scoring method) were compared in evaluating argumentative essays written by Korean high school students. The present study aims to investigate the rater-reliability of the three scoring methods, holistic, analytic, and multiple-trait scoring methods. Scores of the five raters which were obtained from using the three scoring methods were compared. It was found that there were significant mean differences in the three scoring methods. Raters gave the relatively low scores when they used the holistic scoring. Next, the highest inter-rater reliability was found in the multiple-trait scoring. All the three scoring methods showed an acceptable level of reliability above .07. However, raters showed the highest reliability when they used a multiple-trait scoring rubric. Also, high correlation was found among components of analytic and multiple-trait scoring methods, indicating that the multiple-trait scoring rubric can replace the analytic scoring rubric. Finally, raters expressed a favor over the multiple-trait scoring. The result of this study suggests some implications for writing assessment in Korean secondary English classes.