Multiple Choice Questions on Exams

[This is a guest post by Derek Bruff, an assistant director of the Center for Teaching at Vanderbilt University, where he is also a senior lecturer in Mathematics. His book, Teaching with Classroom Response Systems: Creating Active Learning Environments, about the technology known as clickers, was published in February, 2009 by Jossey-Bass. On Twitter, he is @derekbruff. -- JBJ]

A couple of years ago, I was teaching a statistics course for the second time. The first time I had taught the course, I only had 36 students. This time I had 56. Thinking ahead to how long it would take me to grade their exams, which had consisted entirely of free-response questions the year before, I started wondering if I should include some multiple-choice questions on the exams this time around. I felt a little ashamed. After all, instructors only gave multiple-choice exams in those really big classes where free-response exams weren’t possible, right? It’s not like they want to give multiple-choice exams—they’re forced to by the size of their classes.

Then it hit me—putting multiple-choice questions on my exams was exactly what I wanted to do.

Let me rewind a little. A few years prior to teaching this statistics course, I had started teaching with classroom response systems, often called “clickers.” When using a classroom response system, I can pose a multiple-choice question to my students during class and expect each and every one of my students to respond to the question—independently, in fact—by submitting their answers using handheld radio-frequency devices (“clickers”). A receiver attached to my computer collects their response, and the clicker software displays a bar chart showing the distribution of results.

Over the years, I had gotten pretty good at writing multiple-choice clicker questions. I find that the multiple-choice format is particularly useful for engaging and assessing students around the more conceptual material in my math courses. For instance, here’s a question about confidence intervals I’ve used:

Suppose you construct a 95% confidence interval from a random sample of size n=20 with sample mean 100 taken from a population with unknown mean μ and known standard deviation σ= 10, and the interval is fairly wide. Which of the following conditions would NOT lead to a narrower confidence interval?

A. If you decreased your confidence level

B. If you increased your sample size

C. If the sample mean was smaller [Correct]

D. If the population standard deviation was smaller

What makes this question useful is that it asks students to respond using their intuition about how confidence intervals work. Although there are a few numbers in the question to make the situation somewhat more concrete for the students, this is not a computational question. Instead, the question asks students about the relationship between the width of a confidence interval and several other variables.

I feel confident that my students, mostly engineering majors, could answer a computational question on this topic. I’m typically less confident that they have internalized the associated concepts. Asking this clicker question allows me to assess their conceptual understanding and, if the results show that students don’t understand the situation as well as I would like, I can help them think through the topic right then and there during class.

Now, back to the statistics exam. My students and I spent at least a third of every class period working through conceptually-oriented, multiple-choice clicker questions. If I thought this was such a good use of class time, then why didn’t I include similar questions on my exams? More to the point, if I wanted my students to develop conceptual understanding of the statistics in my course, why wasn’t I assessing that? Particularly since I knew exactly how to do so, by asking the same kinds of multiple-choice questions I had gotten good at writing over the years teaching with clickers!

Starting that semester, I started writing my exams so that multiple-choice, conceptual questions contribute about 40 percent of my students’ grades on those exams. Free-response, computational questions contribute the rest of their scores. Often the multiple-choice exam questions are refined or enhanced versions of clicker questions asked earlier in the semester. Writing these questions doesn’t take much longer than writing free-response questions, and, of course, they help me look forward to grading a little bit more!

Have you found that the multiple-choice format is sometimes exactly what you need to assess particular aspects of your students’ learning? How have you used multiple-choice questions to assess more than mere factual recall? Share your experiences in the comments.

Image by Flickr user Mars Hill Church Seattle / Creative Commons Licensed

Comments

1. Andrew Dawes - March 05, 2010 at 11:33 am

This is exactly what I have come to find too. I use clickers in my larger classes (a 90-student optics class in particular), but even for my smaller classes I find multiple-choice questions useful on the exams. I even go one step further and throw in some True/False for conceptual evaluation. My most recent test, an exam for our 11-student Quantum Mechanics class, had about 40 percent multiple choice or true/false questions. Physics tests can very easily become tests of a student's mathematical ability rather than their conceptual understanding. Using a variety of question styles provides a unique way to access conceptual understanding without testing for algebra or calculus proficiency.

The one drawback that I find with m/c or t/f questions is the quantization of the score. They can either be right or wrong so it is easy to lose a lot of points through a few mistakes. To alleviate this I require that they also explain their answer. Each question has space provided for doing so. This explanation allows me to award partial credit if their understanding is only marginally incorrect, and it adds very little grading time since I can still glance at the page to identify right and wrong answers. The wrong answers then get a little more attention to award partial credit if it is deserved.

Great post! I'm glad to see that this approach is working for others too. -Andrew

2. Stephen Schellenberg - March 07, 2010 at 08:21 pm

With enrollments approaching 200+, I long ago gave in to MC-based exams for my GE courses in oceanography. I try to turn this lemon into lemonade in two ways:

First, like Derek, I strive to develop really good multiple choice questions aligned with the student learning outcomes. Post-hoc point-biserial analysis is a useful population-specific tool for identifying such questions, and these questions can then often be varied their structure/response for repeated use.

Second, my students take each MC exam twice during the scheduled exam time: The "individual" mode is proctored under typical test conditions for the first ~2/3rds of the class time. Student enter their responses on scantrons, and then turn these scantrons in (but keep their exam sheet) when time is called. The student then retake the exam in a "collaborative" mode for the remainder of the class time, revising their responses individual or in self-organized groups with reference to their lecture handouts, course notes, the web (danger, danger as I stress to them), etc.

The final "combined" exam score is a 75% weighting of the individual score and 25% weighting of the collaborative score, with the condition that if you collaborative score is lower, you simply get your individual score for the exam. Interestingly, about 5% of my students skip the collaborative portion altogether (said students typically get 90%+ on the individual) and about 5% of my student manage to do worse on the collaborative (which brings up teachable moments as you might imagine).

This approach provides an incentive for students to immediately self-assess and verify their understanding, while the 75%/25% weighting generally doesn't wreak major rank-order changes between the "individual" and "combined" scores. In addition, the "collaborative" portion actually capitalizes upon the MC structure by focusing discussions within the self-organized student groups.

Student like this strategy and it is one of the few times that I can stand in a room and be surrounded by student-centered discussions centered on course topics (think-pair-share with clickers is another). Finally, note that one could readily implement this general approach with clickers, though I haven't taken that step (or potential plunge).

Regards, Stephen

3. Derek Bruff - March 10, 2010 at 12:33 pm

Thanks, Andrew. I'm torn on the quantization issue. On the one hand, MCQs don't let students earn partial credit when used in the traditional fashion on exams. (One could, however, designate certain wrong answers as better than others and thus worth partial credit.) On the other hand, students do have the option of guessing their way to full credit, so I think it's a bit of a toss-up.

Something I've done a few times is to have students identify their confidence level--high or low--in their responses to True/False questions. (I heard about this from a colleague at a math conference last year.) Students earn 6 points for high confidence in a correct answer, 4 points for low confidence in a correct answer, 2 points for low confidence in an incorrect answer, and 0 points for high confidence in an incorrect answer.

The students were surprisingly cool with this grading scheme, given how non-traditional it is. It addresses the quantization problem, I think. Students can earn partial credit on a True/False question, and there's a penalty of sorts for random guessing.

4. Derek Bruff - March 10, 2010 at 12:36 pm

I like this idea a lot, Stephen. It adds a learning component to what is typically "just" an assessment experience. The team-based learning approach to teaching uses a similar technique--individual quizzes followed by group quizzes.

I do something that hits some of the same notes (but definitely not all of them). I give my students the option to submit test corrections after I hand back my exams. They can correct any question on the exam where they lost points, including the MCQs, for up to 1/3 points back. So if a student makes a 70, that's 30 points lost out of 100. They can bring that up to a 70 + 30/3 = 80. For the MCQs, I require the students to not only provide the correct answer but also the correct reasoning for that answer.