At the risk of having my edupunk fellow-traveler card revoked, this morning I’m going to cop to using a course management system. Most days, students in my literature classes will have accessed Moodle, and will have taken a multiple-choice quiz on the day’s reading.
This might not seem terribly helpful. The quizzes are open book, and even if they weren’t, they’re administered online, and so students could cheat if they wanted to, and I wouldn’t know. And a multiple-choice reading quiz about, say, In Memoriam is hardly the royal road to critical thinking.
But it turns out that quizzes are locally pretty useful. Students are better prepared when there’s a quiz, and so discussion goes a bit better. Also, in a literature class, the daily quiz offers a way to make sure they’re keeping up with the reading. Even the fact that I tend to write them easy (survey-level quizzes are plot-only; upper-level quizzes focus on the plot but may include slightly more conceptual questions; at both levels I will sometimes include questions of the form, “what would be a good question here?”) helps, because students don’t get all caught up in worrying about them.
The secret sauce to the quizzes is, frankly, the course management system. Here’s why:
- The online delivery means that the quizzes take zero minutes of class time. I usually teach the MWF (50-minute) schedule, and I can’t be wasting time on quizzes.
- The online delivery also means that students can look at the questions before they read, which, if I’ve thought about the questions enough, can help frame the reading a bit.
- Because the quizzes are multiple choice, Moodle grades ‘em. I’ll repeat that: THE CMS GRADES THE QUIZZES! This is awesome for lots of reasons: 1) the student knows, right away, how they’re doing. 2) I know before class starts whether the class has understood the assigned reading. And, 3) I don’t have to grade them. I’m never behind on them.
- It turns out that the quizzes, easy as they are, reductive as they are, do usefully predict student performance. They’re terrible at differentiating between A/B range students, but they do an awesome job at sorting students who are at risk. Over the course of the semester, there are so many questions that if you miss 1 or 2, it’s no big deal. But if you’re *always* missing 2 out of 5 questions, then it will eventually dawn on you that you’re doing D-level work.
I’ve got lots of assignments that reward creativity, rigorous analysis, brisk writing, and so forth–sometimes all at the same time!–but the very banality of the quizzes, perversely enough, makes them helpful. The quizzes help set the expectation that reading is to be done every single day, and I don’t have to publicly embarrass anyone to establish it.
This is the only example I can think of where technology operates as it would have in the future we were promised, where it serves as a automagical labor-reducer. (“You mean, I can give quizzes, and not have to grade them?”) It’s also the only assignment my students perform online that, purely and simply, replicates a pre-Internet activity.
Do you have a daily assignment? Have you figured out a way to automate it? (I gather that in some disciplines you can actually buy question banks. Alas, I have to craft my own questions.) Does it work for you? Is the whole basis of the assignment evil and wrong? Does it matter what kind of class you’re teaching?
(In a future post I’ll probably admit to the fact that I require students to memorize/recite poems. But not yet.)
Image by Flickr user bdunnette / Creative Commons licensed
Follow Profhacker through your favorite RSS reader: 
Developing online and blended learning programs requires research and collaboration. Learn how top technology companies are partnering with campuses across the country to advance online learning as it becomes an increasingly important aspect of higher education.
11 Responses to Quizzes in an Age of Course Management Software
Jason B. Jones - February 21, 2010 at 10:18 am
In general, I want to associate myself with the comments by Dave and Nels: The multiple-choice quizzes are very low stakes (no more than 10-15% of the grade, depending), and I don’t base exams or anything like that on them. I’m pretty clear-eyed about what they can and can’t achieve.
Also, discipline matters here: These are lit classes, where a big part of the class is just knuckling down and doing the reading. (Esp. in a Victorian lit class, for example, with 1000-page novels.) The quizzes reward students for keeping up.
To Rebecca, I would say that I’ve had similar classes, especially at the gen-ed level. So much depends on the mix of students. It may be the case that you have to work through a couple of examples in class–but then that defeats the purpose of the CMS!
Rebecca - February 20, 2010 at 1:02 am
I teach psych101 at a community college. I see them once/week for 2 hours and 40 minutes. I have to cover 16 chapters in a semester (which would only be 1/week if I could get them to read a chapter before the first class; since most don’t even buy the book until week 2 this is hard). This is the first semester I have tried doing the pre-class chapter quizzes each week. So far it seems to be the best way for me to ensure that they are reading the material and keeping up with it. I use our CMS (blackboard) and a combination of multiple choice and matching questions that can be automatically scored by the system. The test appears as soon as the prior class is over and is available up to the start time of the class that the reading is due. If they don’t take it, they get a zero. I drop the two lowest grades during the semester, but the average of the rest is worth 35% of their grade.
There are two things so far that are really surprising me. First, that the scores are generally mediocre. Despite being open book (not to mention open note, internet, study group, whatever), untimed, and that they can take it up to 3 times in order to improve their scores, many are still doing poorly. I thought this would be easy points, but the class doesn’t think so. Second, the regular test scores are no better either. It’s as though they don’t remember the material a couple weeks later when I give them a test, despite the fact that it contains reworded versions of many of the same questions.
Dave Guinee - February 20, 2010 at 8:43 pm
I agree in general about multiple choice questions with you, Todd, but I don’t in this case. I regularly teach an introductory Greek and Roman mythology course, and it is a literature-based course. I don’t always use this kind of quiz, but I sometimes do, and I think it’s very valuable. I always do what I think Jason does here; the multiple choice quizzes don’t carry nearly the weight of more substantive assignments. Furthermore, I am clear with my students that the purpose of these is simply to be sure that we have a place to start from, that all (or the majority) of the students have at least have the text well enough to know what happened and pick up major plot points. These multiple choice questions are only one part of the reading assignment; I also supply more thought-provoking study questions and often have students doing forum postings where they do close readings (and critique each others’ close readings).
The more I think about it, the better a message I think this sends. We’re saying, you obviously have to know what the text says (at the most basic level) before we can go anywhere, but this isn’t what we spend our class time on. In fact, it’s too simplistic for me to have to take time to grade it.
Nels - February 19, 2010 at 1:18 pm
I give in-class quizzes. Three questions on the reading that are so obvious, they insult the students who read and annoy the students who didn’t because they are so easy. I usually have one question from the start of the reading, one from the middle, and one from the end. They take them on index cards I pass out in the first five minutes of class. And Jason is so right. The discussion on the days we have quizzes is ten times better than the discussion on days we don’t.
I prefer in-class quizzes because they make students show up on time. I tell them they have to be there at the start and cannot make them up at all for any reason. My attendance drastically improved when I added quizzes to class. I, on average, only teach two classes, though, so my grading time is my smaller than for someone like Jason who I believe has a larger class load then I do. I usually get quizzes graded in the few minutes before a meeting starts or while the class is doing some small-group activity.
As for whether multiple-choice questions promote literal learning over other types, that might be true if the quizzes make up a majority of the student’s grade. In my classes, quizzes are 10-15%. The majority of their grade comes from formal papers. I’ve noticed the quality of the papers improves because students keep up with the reading. They tell me that they read the work for my classes before their other classes because “you’re the only one who checks to make sure we read.”
Kaitlin - February 19, 2010 at 11:16 am
For Blackboard at least, I know you can get question banks (and other CMS material) as part of your textbook package. Granted, there will be hundreds of questions, so you’ll have to sort through to figure out if they’re worth your while. Automated tests and quizzes require a lot of work up front, but can save a lot of time in the long run if you recycle them from course to course.
@Eric – There are often options to set a time limit or deadline for a quiz like this. In Blackboard you can set the quiz link to disappear after a certain time, or it will notify you if a student goes over the time limit and you can deduct points manually.
Jason B. Jones - February 19, 2010 at 11:19 am
@KBHC I do have most students in class doing at least some writing (at least theoretically) a couple of times per week. That’s the part where more interesting comments emerge–and, often, leads directly to student-generated exam questions.
@Eric: The quizzes are posted to Moodle at least a day or so in advance. And students must complete them before class. The software enforces this, by closing the quiz 5 minutes before class time, so there’s no chance to take it after. Students who miss the quiz get a zero for that day. Over the course of the semester, missing a day or two won’t kill you–but a pattern of skipping ‘em will.
@Kaitlin: There are banks of quizzes for Moodle, too–my problem is with discipline (fewer in lit classes than in some others) and text selection.
KBHC - February 19, 2010 at 10:41 am
I also use Moodle for my courses and this semester started requiring an assignment to test reading comprehension before my students show up in class. Instead of writing a multiple choice quiz, which would take me a little while, I require my students to write a one paragraph reading summary (and I spell out the contents very explicitly) that they upload to Moodle. I use the Lesson function, create an Essay question page and have their essay count for zero points. Then the second question page is true/false and asks “Did you perform this task truthfully and to the best of your ability?” True is ten points, false is zero points. That way I get Moodle to grade these automatically, while still asking my students to write a little something about the readings.
The other thing I do is I read a subset of the summaries every time. This is for quality control (I go into the gradebook and change grades for crappy assignments) and because I like seeing what they write and what they think. Often I get at least one great discussion question out of it, or I get a sense of what the students aren’t understanding in the reading. Then I modify what I’m teaching that day or what group activities we are doing, to address what the students are saying.
In the end, this works out to less prep for me, an assistant prof in the social and life sciences, yet the quality of conversation in both my classes this semester has been orders of magnitude higher than I’ve had before.
Eric Castro - February 19, 2010 at 10:47 am
So, just to be clear: these quizzes are posted to Moodle and students are expected to complete them before the class meets? That is, they are not doing them during class?
How do you handle students who do not take the quiz? Do they get a zero? How about students who take it after the class meeting?
Bill - February 19, 2010 at 11:44 am
I have long given regular, but not daily, quizzes in my constitutional law classes over facts and other basic concepts in cases. I have slowly moved some of them over to the CMS system we have on campus. The purpose of mine is to get higher percentages of reading among the class and sometimes with a short essay question to prime them for a particular discussion topic. I too have the problem of no test banks, but questions like this are not hard to write. I have found that they do work reasonably well. Our CMS, ANGEL, makes grading short essays easy as well. I can look at all the answers to a given question on the screen, with or without names attached, and grade them in rapid succession. Very fast grading, much faster than paper. I always allow students to drop their worst 1 or 2 grades on quizzes. This avoids what to do when a student is sick or on a college sponsored trip without Internet access. The CMS can do this automagically. (I also have an Excel formula that does it.)
I have thought about everyday quizzes as I sometimes get a bad day on the next class after a quiz. Have you gotten a lot of pushback about daily quizzes?
A couple of other questions: how much are these quizzes worth toward the semester grade? how many questions do the quizzes have?
Todd Finley - February 19, 2010 at 11:56 am
Jason,
The use of multiple choice quizzes dehumanizes the instructor-student-content interaction. And it emphasizes a priority (literal level understanding) about what counts in the readings that I doubt our agile instructor/writer really wants students to internalize.
When we use the multiple choice format, aren’t we reinforcing achievement at the expense of learning?
Alphiekohn.org and FairTest.org, have excellent commentary on why multiple choice quizzes are problematic.
TBF
Todd Finley - April 2, 2010 at 12:47 am
Dave,
The last paragraph of your reply (“We’re saying, you obviously have to know what the text says (at the most basic level) before we can go anywhere, but this isn’t what we spend our class time on. In fact, it’s too simplistic for me to have to take time to grade it”) might be the best argument for the pre-class use of multiple choice quizzes that I’ve encountered. Furthermore, my understanding of the research on testing is that stress is reduced when the point value of quizzes is reduced and when many opportunities to succeed are provided.
However, correlating multiple choice quizzes with efficiency is a myth. To be useful, they take a long time to construct. When I have employed a multiple choice quiz provided by a textbook company, the questions are typically ambiguous, syntactically misleading, and/or just plain trivia. As a result, I have to construct tests that actually measures students’ comprehension instead of their testing skills. All this eats up time and shifts their reading attention towards concrete, testable, chunklets of information, often at the expense of more difficult conceptual knowledge. When students challenge “wrong” answers, more time is lost.
More time is lost when the instructor changes course texts.
I am persuaded by your argument to a point. And I might agree that pre-reading quizzes are more suitable for some content over others. Ultimately, my preference is to spend more time creating assessments that measure my students’ ability to use content knowledge and skills to make decisions, solve problems, and contribute to improving the world. Tinkering with multiple choice gate-keeping feels like I’m letting down my discipline, when my time can be used for designing rich experiences; meanwhile, learners’ valuable time can be used to master the content, rather than jumping through hoops and demonstrating readiness that can be faked (see How to Pass a Multiple Choice Test without Studying: http://bit.ly/acLKGj) . Of course, the last paragraph can be dismissed as idealistic. Yet, how we spend our time and how we focus students’ energy signals our priorities.