Computers can now grade students' writing

South Dakota is one of several states that has tested essay-grading software.  Officials there decided against using it widely, saying feedback was negative.

Not all districts had the same experience.  Watertown, S.D., students are among those who now have their writing-assessment tests scored by computer.  Lesli Hanson, an assistant superintendent in Watertown, said students like taking the test by computer and teachers are relieved to end an annual ritual that kept two dozen people holed up for three days to score 1,500 tests.  "It almost got to be torture," she said.

Some 80 percent of Indiana's 60,000 11th-graders have their English assessment scored by computer, and another 10,000 ninth-graders are taking part in a trial in which computers assess some routine written assignments.

Stan Jones, Indiana's commissioner of higher education, said the technology isn't as good as a teacher but cuts turnaround time, trims costs and allows overworked teachers to give written assignments without fearing the workload.  "This (allows) them to require more essays, more writing, and have it graded very painlessly," Jones said.

Software can also remove a degree of subjectivity.  "It's fairly consistent.  Different teachers grade different papers differently." said Keith Kelly, 21, of Cleveland, one of Brent's sociology students.

The software is not flawless, even its most ardent supporters admit.

When the University of California at Davis tried out such technology a couple years back, lecturer Andy Jones decided to try to trick e-Rater.  Prompted to write on workplace injuries, Jones instead input a letter of recommendation, substituting "risk of personal injury" for the student's name.

"My thinking was, 'This is ridiculous, I'm sure it will get a zero,'" he said.  He got a five out of six.

A second time around, Jones scattered "chimpanzee" throughout the essay, guessing unusual words would yield him a higher score.  He got a six.

In Brent's class, sophomore Brady Didion submitted drafts of his papers numerous times to ensure his final version included everything the computer wanted.  "What you're learning, really, is how to cheat the program," he said.

Work to automate analysis of the written word dates back to the 1950s, when such technology was used largely to adjust the grade level of textbooks, said Henry Lieberman, a research scientist at the Massachusetts Institute of Technology.  Before long, researchers aimed to use such applications to evaluate student writing.

SAGrader, like other programs, needs significant prep work by teachers.  For each of the four papers Brent assigns during his semester-long course, he must essentially enter all the components he wants an assignment to include and take into account the hundreds of ways a student might say them.

Part of one assignment for Brent's class was for students to pick a crime and explain how it fit into sociologists' categories.  Brent had to key in dozens of words in order to ensure all types of transgressions would be identified.

What a writer gets back is quite detailed.  A criminology paper resulted in a nuanced evaluation offering feedback such as this: "This paper does not do a good job of relating white-collar crime to various concepts in labeling theory of deviance."

Brent — who earned a postdoctoral degree in artificial intelligence and is also an adjunct professor in the computer science department — said the software may have limitations, but allows teachers to do things they weren't able to do before.

Before Brent wrote SAGrader, a part of his broader data-analysis program Qualrus, he only gave students multiple-choice tests.  "Now we can focus more," he said.  "Are they making a good argument?  Do they seem to understand?  Are they being creative?"

MORE FROM TECH AND GADGETS

Tech and gadgets Section Front

Last-minute Web sites to save your holiday Hey, Tiger Woods: Why so dumb about tech? Teen Internet addicts more likely to self harm Will CrunchPad re-emerge with new name? AP source: Apple in talks to buy Lala.com Microsoft, Yahoo seal search deal 10 iPhone apps for the holidays that sparkle Digital photo frames not always a ‘snap’ to use Groom creates viral storm tweeting from altar Jury gets case of blogger accused of threats Tech and gadgets Section Front   Add Tech and gadgets headlines to your news reader: