next »

[educator1]

Why the inflated title on this article?  This sounds like a highly organized approach to TA grading on the one hand and the automation of that chore on the other. I don't get the controversy. 

http://chronicle.com/article/To-Justify-Every-A-Some/128528/?sid=wb&utm_source=wb&utm_medium=en

It is true that professors are not very consistent both within and, especially, between themselves when it comes to grading. If professors set the rubrics and review the inter-rater reliability, what is wrong with hiring others to do the actual chore of grading or even automating it?

[prytania3]

It ends up being teaching to the test.

When I first started my job, we had an exit exam in Composition, and two other people graded your class.

It became very standardized and very stressful.

[polly_mer]

I'm ok with mass grading of computations.  Anyone qualified to do the assignment who has my answer key with my standard for partial credit can grade those.  However, in my class, you will use my rubric.  As an exit exam, sure, use whatever you like, but I want to see those questions before they are administered and see the rubric being used.

I'm not ok with the suggested robo-grading of essays.  Let's start with the fact that most grammar checkers are wrong to some degree.  We can then move on to the meat of the subject where students will often use key words incorrectly or skip the terminology to write something perfect, but may not be acceptable to the machine.  I don't know how many of my students have ever done robo-graded essays, but I can pick out students who are used to being graded by a rubric of "included discussion of X" rather than being graded by having a clue about X and making a logical point.

[concordancia]

On the other hand, if turnitin's new grammar grader turns out to be accurate, it could be an amazing tool for streamlining essay grading. As I mentioned to a friend earlier today, I do not think it could be used in developmental courses, where the students would be entirely overwhelmed with the feedback.

[polly_mer]

On the other hand, if turnitin's new grammar grader turns out to be accurate,

Word is terrible for grammar checks.  What is the expectation that Turnitin got the grammar right?

[prytania3]

On the other hand, if turnitin's new grammar grader turns out to be accurate,

Word is terrible for grammar checks.  What is the expectation that Turnitin got the grammar right?

That's because WORD grammar was written by someone whose second language is English.

I tell my students, "Don't use the grammar check!"

[polly_mer]

On the other hand, if turnitin's new grammar grader turns out to be accurate,

Word is terrible for grammar checks.  What is the expectation that Turnitin got the grammar right?

That's because WORD grammar was written by someone whose second language is English.

I tell my students, "Don't use the grammar check!"

I won't speculate on the reason that WORD grammar is bad, but I frequently get suggestions that are clearly this-word-triggers-a-rule instead of based on how the sentence uses the word.

[mountainguy]

I agree with Polly that robo-grading of essays is a terrible idea. The technology is still in its very early stages, and beyond that, it seems very Cyborgish to me.

That having been said, I do approve of efforts to use behind-the-scenes graders to improve standardization of grades, as long as it's done in a reasonable manner. My own experience with teaching a standardized course has been that there needs to be consistent departmental support for such a system to work; if instructors are thrown to the wolves to deal with angry students or expected to teach concepts/units they can't explain, the atmosphere will quickly become poisonous. But when done correctly, it increases efficiency.

[shamu]

Let me make it clear, I do not have a problem with automatic grading, in principle. Actually, any grading that does not involve thinking can be robo-graded (like multiple choice, matching, or filling in blanks). However, when thinking is needed, I am more hesitant to just hand grading over to a stranger or a machine. My impression is that the robo-grading described in the article is purely a cost cutting measure. 

It is true that professors are not very consistent both within and, especially, between themselves when it comes to grading. If professors set the rubrics and review the inter-rater reliability, what is wrong with hiring others to do the actual chore of grading or even automating it?

Consistency does not equal quality. In principle, "objective" grading is best. as for machine grading, I am not sure there is an algorithm out there that can grade content well (such as how good the narrative is, how solid the line of reasoning is, and so on). As for "objective" outside human graders, as long as they are as qualified (or better qualified) and PAID than the instructor, no problem. My feeling is that in order to get the graders to be "consistent", they just have them grade basic style. That is fine if the course is a rudimentary writing class.

I also think that most professors are quite consistent in their grading. Granted, there may be less consistency across different courses, but they are different courses. I would like to see results from actual studies that clearly demonstrate that robo-grading is BETTER (not just "more consistent") than grading done by instructors who actually teach the course.

This sounds like a highly organized approach to TA grading on the one hand and the automation of that chore on the other. I don't get the controversy. 

1. Having re-read the article, it sounds like the main reason for robo-grading is cost cutting and not ensuring "quality" and "objectivity".
2. I am also not a fan of pawning everything off on TAs, BUT, the TAs are at least aware of what the instructor is teaching (or should be). A stranger who gets a bunch of papers and is paid by the paper
3. Graders should be AT LEAST as qualified as the average professor and PAID accordingly. Until and unless the graders are treated as valuable employees of the institution
4. I would really like to see the average grades given by outside graders. I also want to know how long a harsh grader is kept. After all, one of the arguments is that traditional grading involves tremendous grade inflation ... an issue that should not plague outside graders.
5. Is there a study demonstrating that graduates of a program that uses only robo-signing are much better at writing than their non-robo-graded peers?

So, it is possible that the model described in the article is better than traditional essay grading IF:
1. The main driver is not cost-cutting.
2. The grader is very familiar with what the instructor is teaching.
3. The grader is very well compensated for the work and is AT LEAST as qualified as an instructor should be.
4. The grades given by the "objective" graders are tougher than grades given by instructors. 
5. The data should back up the claims. The results should speak for themselves.

[grasshopper]

On the other hand, if turnitin's new grammar grader turns out to be accurate,

Word is terrible for grammar checks.  What is the expectation that Turnitin got the grammar right?

That's because WORD grammar was written by someone whose second language is English.

Is that true?

[treehugger1]

I agree with polly_mer and mountainguy that the robograding is the most problematic aspect of this trend, but for a reason that they didn't mention.

Even if the software were more accurate than it is now, robograding would still be a bad choice. The whole point of writing is communicating with another person! Yes, even on exams. I repeat, that is the whole point of writing!
Not proving that one knows something, has gotten the "right" answer.

The TA's were way far off on inter-rater reliability," she says, referring to whether different people scored the same test consistently. The graduate students became fatigued and made mistakes after grading several tests in a row, she told me, "but the machine was right-on every time.

Well, I think we're all bright enough to see that there more than one way of solving this problem.

[concordancia]

I agree with polly_mer and mountainguy that the robograding is the most problematic aspect of this trend, but for a reason that they didn't mention.

Even if the software were more accurate than it is now, robograding would still be a bad choice. The whole point of writing is communicating with another person! Yes, even on exams. I repeat, that is the whole point of writing!
Not proving that one knows something, has gotten the "right" answer.

The TA's were way far off on inter-rater reliability," she says, referring to whether different people scored the same test consistently. The graduate students became fatigued and made mistakes after grading several tests in a row, she told me, "but the machine was right-on every time.

Well, I think we're all bright enough to see that there more than one way of solving this problem.

The robograding that I referred to is about marking their grammar mistakes (and turnitin's version claims to link students to appropriate grammar lessons - like I have the time to do that on every paper!), leaving me to focus on the content. I don't like sending the message that they are really that different, but I do have fantasies of never having to write anything related to "subject-verb agreement" ever again. In my dream world, I would run their essays through RoboGrader, then sit back and comment on logic, argument and organization, relishing the fact that the grammar has already been taken care of.

[treehugger1]

Sorry concordancia, I must have missed your previous post. Yes, that makes sense and is analogous to robograding of computer science assignments (a practice of which I was the happy beneficiary waaaay back in the 1980's).

If I'm not mistaken, however, the article referred to Robograding of entire essays (in biology) -- content included.

[oldfullprof]

1.  I'm totally against standardization.  Courses and professors are individuals.

2.  I was in a general education course training once with twenty other faculty, where 18 of them (all young) could not identify very good writing if it had a number of trivial grammatical errors in it.  This is terrible, and there's no way I'd trust anyone else to do my grading.  Sorry.  Never.

[mended_drum]

On the other hand, if turnitin's new grammar grader turns out to be accurate,

Word is terrible for grammar checks.  What is the expectation that Turnitin got the grammar right?

Even if they get it right, it won't work at my SLAC, as it is perfectly possible to write an essay with no grammatical errors that still receives a failing grade.  There's a whole lot more to write effectively than grammar.