Reliability and Validity Assessment of Automated Essay Scoring Systems on Graduate Students’ Writings

【Abstract】The importance of correction and feedback in English writing teaching cannot be underestimated in terms of students writing ability， interest， even the quality of English teaching. With the rapid development of technology， computers are widely used in English writing teaching， such as the application of automated essay scoring （short for AES） systems. The author of this paper is going to research reliability and validity of assessment on graduate students writing assignments by using Pigaiwang. Through data analysis， it can be found that there are reliability and validity of the assessment in graduate students writing assignments by using Pigaiwang to some extent.

【Key words】English writing； assessment； validity； reliability； Pigaiwag

【作者簡介】谢璐，海南大学。

1. Introduction

English Writing， as an important language output， is a very useful way to test students comprehensive abilities and cognitive levels. Along with the rapid development of science and technology， intelligent software that could reduce their evaluating and grading load appeared， which leads to automated essay scoring （AES） system. The author of this paper conducted a research on feedback in Graduate English writing based on automated writing evaluation system by figuring out its influence on students English writing from the perspective of reliability and validity.

By reliability， it means that the consistency of scoring by two or more scorers. unclear scoring criteria， figure， carelessness. For validity， there are 3 types of validity mentioned in the book of Brown： content validity， face validity， construct validity. Rezaei and Lovorn （2010） pointed out that using rubrics may not improve the reliability or validity of assessment if raters are not well trained on how to design and employ them effectively. Gerde and Bingham （2015） showed that teachers vary widely in their material and instructional supports of writing in preschool classrooms. Attali （2006） came to the conclusion that feedback provided by the AES system can help students improve their writing level by comparing the first draft of the students and the revised article.

2. Data analysis of writing assignments

2 college English teachers of university were required to give a score on students writing assignments based on content， organization， grammar， vocabulary and appropriateness. 10 graduate students from different majors of Chinese university were selected to write an essay on appointed topic. They were required to write an essay within 300- 350 words in 40 minutes. The full score was 100 points.

After finishing writing the assignments， 2 selected teachers were required to give them scores.

It can be easily seen from the above table that scores given by Pigaiwang are higher than teachers， the mean score of Pigaiwang is 88.85， teacher 1 is 86.25， teacher 2 is 86.60. The higher score Pigaiwang gives， the higher score teacher will also give. The gap between standard deviations are also very small. Even though there are not big score differences between Pigaiwang and teachers， scores given by Pigaiwang are higher than teachers overtly. It can be described that there are reliability and validity of assessment in graduate students writing assignments by using Pigaiwang to some extent.

3. Conclusion

According to the research， it can be found that Pigaiwang can always provide timely feedback on for students writing assignments as soon as they submit them online. Through the data analysis， it can be summarized that there are reliability and validity of the assessment in graduate students writing assignments by using Pigaiwang. However. There are still some disadvantages by using it. For example， it is weak in evaluating logic and content and more.

References：

[1]Attali，Y.Automated Essay Scoring with E-rater[J].Journal of Technology，Learning and Assessment，2006，4. Print.

[2]Rezaei，A.R.，Lovorn，M.Reliability and validity of rubrics for assessment through writing[J].Assessing Writing，2010，15：18-39.