Advantages of Automated Essay Scoring (AES) Systems
One of the biggest advantages of an AES is that a student receives quick feedback. An AES provides feedback to a student almost instantaneously with submission in most cases. Especially in large classes the speed of feedback is far superior to that of human graders. This is one reason why so many computer science instructors have created homegrown AES systems to grade introductory computer programming courses. Grading of programming is tedious and can be handled much more quickly and efficiently by an AES. Quick feedback is vital and often listed as a best practice for teaching and often recognized as being essential in student motivation to learn (Barker, 2011). In distance education where interaction with an instructor may be minimal or sporadic the use of an AES can not only score essays but tutor the student as well (Duwairi, 2006).
Besides speed of feedback an AES system can also provide consistency in grading, cost-savings for the educational institution, time-savings for the instructor, and reduced error in scoring. Time and date of submission is recorded automatically, and since computers can only be objective no personal bias in grading is possible (Olivier,
Herson & Sosabowski, 2001). Another AES benefit is its ability to act as a plagiarism detector. In one study an AES discovered an extremely high rate of plagiarism finding that 98 out of 712 assignments were copied from another student’s work (Cheang, Kurnia, Lim & Oon, 2003).
We have all encountered students who become discouraged or belligerent when subjected to criticism even when it is constructive. Students may be more open to such criticism if it is delivered impersonally through an AES. An
AES also encourages students to revise their work before submitting it for final grading. In one study students who used an AES wrote three times as many words on an essay as students who did not use an AES (Giles, 2011).
Finally, many Net Generation students enjoy the gamification aspect of utilizing an AES. Students treat the AES system as a video game “in which doing well involves redrafting work to get a higher score” (Giles, 2011, p. 22).
Disadvantages of Automated Essay Scoring (AES) Systems
While there are many advantages to employing an AES there can be some challenging problems involved in their use. Some students do not like the idea of a computer scoring their work. Lai (2010) found that students preferred their writing be evaluated by a peer rather than by an AES. Students often worry that an AES cannot understand novel ideas and concepts, or properly grade answers that were not part of its training (Landauer, Laham & Foltz,
2000). Barker (2011) found that as AES feedback increased so did the amount of challenges by students about their grade. Another issue, especially with homegrown AES systems is limited feedback. For instance, when used for programming grading the student may only be told that their code was incorrect. Blayney and Freeman (2008) used a homegrown AES to score basic accounting problems with mixed results. Some students were able “to selfdiagnose specific mistakes, but many “novice learners were overwhelmed by having the onus to identify their current error and the improvement advice” offered by the AES (Blayney & Freeman, 2008, p. 165).
Early AES systems such as PEG were susceptible to cheating as writing long essays tended to result in a higher score. Critics often claim that AES systems cannot evaluate the subtleties of good writing. A computer program was incapable of measuring the most important qualities of fine writing such as content, organization and style (Hearst,
2000). In one study, instructors abandoned use of an AES system for this very reason. When 10 of 33 students requested a human re-grading the result was an average increase of 24% in their scores. The instructor found that
“machine readers could not detect subtleties of writing such as irony, metaphor, puns, connotation and other rhetorical devices” (Byrne, Tang, Tranduc & Tang, 2010, p. 35). Even AES developers admit that their systems are incapable of scoring English composition assignments or creative writing; the systems work best when scoring factual knowledge (McCollum, 1998).
There can also be technology issues with use of an AES. Because the student is submitting their essay online the speed and reliability of the Internet connection is an important consideration. There is also the very real problem of computer anxiety. Lai (2010) found that over half of the students expressed confusion with the complex AES interface and that the “slow speed of the Internet connection frequently made them impatient and uncomfortable (p.
443). While most Net Generation students are familiar with and enjoy using computers there is still a substantial minority who fear and loathe technology. Some students find the frequent computer interaction required by an AES to be dehumanizing (Lai, 2010). Lastly, most AES systems require a long period of training before they can be employed. “In order to train a machine learning system, a corpus of holistically scored essays needs to be collected, so that it can be used as training data for the system” (Santos, Verspoor & Nerbonne, 2012, p. 5).