The effects of raters familiarity with test takers L1 in assessing accentedness and comprehensibility of independent speaking tasks

Abstract

This study aims to answer the question: does different degree of raters familiarity with the test takers L1 (Korean) affect how they assess accentedness and comprehensibility? Speech data was collected from four Korean undergraduate students and a total of 13 raters participated in the study. The raters are classified into three groups depending on the level of proficiency of test takers L1 and years living in Korea: three of the raters are non-native speakers of Korean, five of them are bilingual speakers, and five are Korean native speakers. They were asked to rate the speech data of approximately 45 seconds for their accentedness and comprehensibility. The group of native speakers gave the lowest scores for accentedness and comprehensibility whereas the non-native speaker raters gave the highest scores for both. The bilingual rater scores fell in between the two groups: they were relatively lenient compared to the native speaker raters but harsh compared to the non-native speaker raters. There was a gradient-like result of the scores depending on the level of familiarity with the test takers language. This study suggests that even a minimum level language familiarity can be a source of bias for there is a correlation with the severity in scoring.