Predicting L2 Writing Proficiency with Computational Indices Based on N-grams

Abstract

Linguistic features that are indicative of higher writing proficiency levels can inform many aspects of language assessment such as scoring rubrics, test items, and automated essay scoring (AES). The recent advancement of computer algorithms that automatically calculate indices based on various linguistic features has made it possible to examine the relationship between linguistic features and writing proficiency on a larger scale. While the ability to use appropriate n-grams – recurring sequences of contiguous words – has been identified as a characteristic differentiating between proficiency levels in the literature, few studies have examined this relationship using computational indices. To this end, this study utilized the Tool for the Automatic Analysis of Lexical Sophistication (TAALES; Kyle & Crossley, 2015) to calculate eight indices based on n-grams from a stratified corpus consisting of 360 argumentative essays written by Korean college-level learners. First, the indices from the training set of 240 essays were used to design a multinomial logistic regression model in order to identify indices that are significant predictors of writing proficiency levels. Subsequently, the regression model was applied to a test set of 120 essays to examine whether the model could be used to predict the proficiency levels of unseen essays. The results revealed that the mean bigram T, mean bigram Delta P, mean bigram-tounigram Delta P, and proportion of 30,000 most frequent trigrams indices were significant predictors of proficiency levels. Furthermore, the regression model based on eight indices correctly classified 52.5% of essays in the test set, demonstrating above-chance level accuracy.