Developing Theory-Based Diagnostic Tests of English Grammar: Application of Processability Theory

Abstract

An important consideration for developing diagnostic language tests is whether these should follow a particular course of study, or whether students and teachers are better served with theory-based tests. One particular grammar-acquisition theory that has garnered recent attention from language testers and SLA researchers is Manfred Pienemanns (1989) Processability Theory (PT). The Rapid Profile diagnostic test has already been developed based on this theory, and is currently in use

one substantial limitation of this test, however, is that it is a speaking test, and therefore difficult to administer in foreign countries. Furthermore, this type of testing requires language testers with either native or native-like proficiency, which may be restrictive in countries such as Korea, China, and Japan. Researchers have developed other tests with different task types using PT. These tend to be productive tasks

however, two notable exceptions are those developed in Norris (2005) and Chapelle et al. (2010). Those studies employed PT to develop university-level placement tests appropriate for computer-based testing, with task types that attempt to imitate production (writing). These tasks can be fine-grained and allow for more control over the contexts that students are given, which well suits PT, and may be more accessible for foreign English language teachers in situations where there are few native English speakers. On the other hand, there are limitations

the test items may not test what they purport to, which leads to a false positive, indicating full acquisition on a grammar point not yet acquired. Another possibility is that the contexts are insufficient to show acquisition for another reason, such as topic unfamiliarity

in that case, the test would show a false negative, suggesting no acquisition when a grammar point has, indeed, been acquired. The test developed for this study incorporates both types of tasks described above: writing and blended. The writing task is a story-telling task based on six pictures, designed to elicit the same types of grammar tested on the second half of the test. The grammar points are similar to those tested in Chapelle et al. (2010), with the exception that there are fewer, which accommodates context requirements while keeping the test at a reasonable length. The test was piloted twice to junior high school students from Korea, and adjustments were made before the final test was developed. This test was given to 200 Korean junior high school students. Students and teachers also received diagnostic feedback based on the test results. Qualitative and quantitative analyses were done on the results to analyze similarities and differences between the two task types and diagnostic information they offered. The results suggested that the two task types performed similarly, in that they both showed implicational hierarchies comparable to those proposed in PT, but the blended-type tasks showed a tendency toward being less productive. Implications for diagnostic test designs are discussed.