Last week, academic scandal-mongers were treated to an unusual spectacle at the University of Central Florida: What at first looked like a juicy incident of mass cheating turned out, on closer inspection, to also point up a less-attractive aspect of contemporary pedagogy: the reliance on publisher-provided test banks for exam questions. If this story is news to you, Robert Talbert has two great posts exploring it here and here. (The role of YouTube in this whole affair–both in establishing the professor’s ethos and in facilitating a student response–is particularly interesting.)
It’s hard to defend faculty who rely exclusively, or even largely, on publisher’s test banks–although if you ask around on campus, you can probably find some folks who do this. (The fact that some publisher’s test banks are crafted to plug directly into prominent LMSes, such as Blackboard/Vista, is worth some reflection.) At the same time, it’s not immediately clear that the faculty are entirely to blame. Some departments may well turn to such publisher-provided test banks as a way of responding to ever-heightening demands for assessment, and of course many part-timers are (explicitly or subtly) steered into such practices. (And, as Robert mentions in his second post on this story, the local context at UCF–often lauded as a model of cost-cutting educational innovation–is not irrelevant.)
Besides obvious benefits such as convenience, test banks do offer an appealing legitimacy to exam questions. Many students complain, or, at least, faculty worry that students will complain, that their final exam was unfairly arbitrary or idiosyncratic, and doesn’t really reflect the student’s knowledge of the subject. Outsourcing exam writing to textbook publishers appears to solve this problem: an exam drawn up by the publisher of the course textbook will (hopefully!) reflect that book and its material accurately. But the disadvantage of this process is pretty clear–it reduces the faculty member to an appendage of the textbook. Indeed, on such a model it’s a bit hard to understand what the need for PhDs really is.
A more interesting alternative to outsourcing the production of exams is to crowdsource them. My wiki-based courses feature student-written exams:
I then give students a week to comb through their notes and their books to come up with passages for identification, short answer questions, and essay exams. The deal I always make is that if the students come up with an adequate number of smart questions, then I’ll draw the exam entirely (or close to it) from their questions, and will usually post it as a study guide a day or two in advance of the final.
What’s nice, and sometimes terrifying, about this approach is that the resulting questions usually do genuinely reflect the class’s work. That is, it quickly becomes clear what your students will be taking away from your class. Further, when the students collaborate in this way, they both have to do the reflective, synthesizing work of question-writing (which is better than cramming) and to come to an implicit agreement about what our course was about.
I like this assignment, and its wikified class notes component (background: here, here, and, from the pre-ProfHacker days, here), a great deal, but will freely confess that it doesn’t meet one of the needs of test banks: it does nothing to reduce the amount of work in a given semester. The point of these new technologies, though, ought to be to help us create pedagogically interesting courses, not deliver someone else’s content more cheaply.
Have you experimented with writing exam questions? Let us know in comments!
Image by Flickr user victoriapeckham / Creative Commons licensed
Follow Profhacker through your favorite RSS reader: 
Developing online and blended learning programs requires research and collaboration. Learn how top technology companies are partnering with campuses across the country to advance online learning as it becomes an increasingly important aspect of higher education.
15 Responses to Exam Questions: Outsourcing vs. Crowdsourcing
matt_l - November 29, 2010 at 4:32 pm
Yes! I have ‘crowed sourced’ midterm and final exam questions from my classes. I typically do this with upper division courses I teach junior and senior level history majors. I do the following:
About a week before the exam I will break them into small groups to draft a list of five terms plus a couple of essay questions. I ask them to try and develop two types of essay questions: i) a narrowly focused topic type essay; ii) a broad synthetic essay that ties together multiple topics or time periods.
I collect each group’s selections, throw out duplicates, edit the essays for spelling and coherence, and post the results as a study guide on D2L. Later on I will choose the essay questions and terms from that study guide.
The negatives: Students tend to write essay questions about what happened. I want them to think about why something happened or what changed and the consequences of that change. So I end up editing the essay questions into something I think is appropriate. Some students mistakenly think that since they have the exam questions ahead of time, it will be easier to write the essay. They don’t realize that the expectations are higher as well.
The positives: it encourages students to work together in productive ways. They do a credible job of reviewing the most important aspects of the class. I think they more likely to review the material together, outside of class, when they have already been working in the same group during class.
A note of caution: I do not think this works with Freshmen and Sophomores. There is too wide of a disparity in skills and motivations, and this makes group work trickier.
prillva - November 29, 2010 at 4:35 pm
I believe tests should allow students to show what they’ve learned, and I’ve almost always had students propose questions for upcoming tests. In discussion courses, my students know I come to class with questions to pose and passages to discuss, but I always let students ask questions and make comments on the readings first. There have been many times when the students have raised all the important issues and asked all the questions without any prompting or coaching from me.
I don’t know that I’ve ever given a wholly student-written test though.
johnbarnes - November 29, 2010 at 6:27 pm
In my humanities classes the lectures and discussions overlap the textbook relatively little; the students are supposed to know the material in both, and I only cover what’s in the textbook if there are questions about it. To deal with the perpetual loud “Will this be on the test?” whine that students use to silence other students who become interested or ask questions, I keep a pile of blank notecards, and when a student asks an interesting question or raises an interesting issue, I make a note of what it was and where the discussion went. Then I draw at least half the questions from those cards. (Often I get good questions about the reading and those are of course eligible too). Those who attend and review their notes do well; those who read the book at home, don’t.
eileenqueen - November 29, 2010 at 7:28 pm
I’m so glad I checked in today! I can use johnbarnes’ notecard technique. Thanks! I love the idea of formalizing the creation of questions, Jason, Matt_l, and prillva! Maybe not the essay questions since those can go in so many directions based on student background (good point). I’ve been reading about how having students revisit the information in more than one way increases their long-term recall of that information. Pressure to use test banks is real, and time constraints are even more real, but there ought to be ways I can do this. Thanks all!
velvis - November 30, 2010 at 8:44 am
My course involving almost exclusively freshmen also involves them presenting the bulk of the course materials: to get them used to presenting, selecting materials based on course goals, find interesting ways to present etc (it’s an intro to ed class) I do require that they involve discussion questions in their presentations.
These have become my final exam. Because we sit in large tables there are multiple versions of the test but no one wants to answer 13 essay questions…so they get mixed and jostled, which was the most difficult part of writing the exam.
tribblek - November 30, 2010 at 9:02 am
I am forever second-guessing my own techniques. It is very gratifying to discover that others share my enthusiasm for student-driven exams. I do a few chapter tests (more accurately, quizzes) where I ask the students to submit “smart” questions in advance. I let them know that I will be creating the tests exclusively from their questions (assuming their questions are reasonable), allowing for my own editing. Each student generates several questions for each section, giving me many questions from which to choose for the actual measurement.
I find that it forces the students to read the text a bit more critically, and to identify the main points as they read (something they need practice doing). Additionally, many students seem to really enjoy “playing professor” – gaining a hint of what it is like to generate exam questions.
Thanks for this article.
jmittell - November 30, 2010 at 9:14 am
An experiment I’ve tried a couple of times to mixed results is to have the final exam of a class (which was about digital media) be collaboratively written via a wiki (see it here if you wish). I posted a list of keywords that the course covered on a class wiki, and students were required to create a page for each term, offering a definition, context, significance, and relevant examples of the term, citing relevant sources. Then I chose a random selection of terms to grade as I would short-answer essay questions, giving the class a grade based on their collaborative writing. I also made the Grading Guidelines page editable, enabling students to collectively decide to what degree I spread the class’s grades based on how much each member contributed to the wiki.
The strengths of this assignment was that the form of the exam mirrored the content of the course quite well. Via the wiki software, I could track student contributions at a granular level. In theory, it facilitated collaboration and active learning in a way that the individual final exam rarely does. The downsides: it was very hard to get students to actively work on the site prior to a day or two before the exam, leading to a frantic dash that was made more stressful by some software glitches. This last-minute approach led to clear differentiation among students used to planning & time management and those who were not, and the diligent were motivated to carry the slackers. And the temptation to cut/paste from Wikipedia or other sources is quite strong (especially the closer to the deadline it gets) and really hard to police.
Has anyone else found good ways to collaboratively give exams like this?
halavais - November 30, 2010 at 9:51 am
Yes, absolutely crowdsource the exam questions, if you have to do exams.
I think the dismissal out of hand of bank questions misses the point. There is nothing wrong with sharing educational resources, including good exam questions. The issue is with sharing them selectively, and assuming (how exactly?) that students are somehow a different class of scholar who shouldn’t have the same access to those questions. In other words, yes to question banks, no to (putatively) secret question banks.
But I have to say, what this points to more emphatically is the ineffectiveness of exams. This is particularly true of multiple guess exams, which I have managed to avoid for the last several years. But I think it may be true of any kind of exam. Authentic (and even stealth) assessment is where it’s at, but requires a radical change in teaching style for many people–myself included.
This is the first semester in many where I planned a final exam. Last night, my students talked me out of it, suggesting better alternatives. Don’t just crowdsource exam questions, crowdsource assessment. Tell the students why you are giving an exam–what your objectives are–and be prepared to be surprised at how effective they are at thinking about their own learning, particularly when motivated by not wanting to take an exam that they think (often correctly) doesn’t serve their learning needs.
idajones - November 30, 2010 at 10:09 am
Oh no, not the “A” word! (Assessment)
I have also tried student-written exam questions. I’ve combined those questions with questions from test banks and my own questions. It worked reasonably well-but those students were juniors and seniors.
I’ve used wiki development as part of a class project. Although the collaboration was there, I had significant problems with plagiarism (and this was in a graduate law and ethics class in which the first assignment was an online discussion on plagiarism!)
I agree that exams should be a small part of assessment. I constantly struggle with figuring out the “best” assessment method. I talk about it more in my blog (http://idajones.wordpress.com/)
I haven’t tried the exam writing exercise with sophomores, but it is worth pursuing. I may add it as a component this semester, for the final exam.
Thanks everyone!
pivoine66 - November 30, 2010 at 10:13 am
I shudder to think what would happen if I tried this in my community college classes. My students have such a range of abilities — returning students with degrees and a high degree of sophistication, 16-year-old high school students working for AP credit, ESL students who can barely write in English, and lost souls who aren’t sure why they’re there — that group projects, never mind wiki test-writing, are virtually impossible. The weaker students let the stronger students do all the work. A high proportion of the students I teach show very little in the way of developed critical thinking skills and just want to be told what to do to get by. As a teacher, I am in the classroom to create learning challenges that take them out of their comfort zone. If wiki exam writing becomes broadly acceptable, you can bet we in the community colleges will be crowded onto the bandwagon, too, just so we look more like “real” college, regardless of our students’ actual needs.
jeblank1 - November 30, 2010 at 10:49 am
When teaching Human Skeletal Biology with Lab at a State University, I have had students develop lab quiz stations since the late 1960′s.
The Lab Practical Exam is 40 stations which students rotate through in 45 minutes. I ask students to propose two different question types: What everyone should know! What separate A from B students!I use 10 of each category in each practical exam (I provide the remaining 20 stations) and make the list of all proposed student questions available (originally in printed from the Library Reserve Room, now on the Web). In general the student questions were well conceived and frequently more difficult that what I would select.
ksledge - November 30, 2010 at 11:35 am
I like this idea and have done it myself a bit. The only problem I find, though, is that students tend to come up with memorization-based questions. (Seems similar to the issue someone else posted above in which students asked “what happened” essay questions.) Even for multiple choice, I prefer questions that go well beyond memorization. But overall I think it’s a great practice for the reasons listed by the author. Moreover, it’s a way to get the students to study the material.
creamcity - November 30, 2010 at 11:55 am
For years, I have had students draft possible exam questions, which works well to promote (in the buzzword I was taught for this) their “ownership” of their own assessment, rather than seeing it as something inflicted upon them. This even has worked well in the lower-level courses, although the range of responses, of course, was not as competent — but some, a sufficient number, were.
I also created a last question, a last chance to do this, on exams: I ask “what did you study that is not on the exam, and that you wish had been asked” — and students love it. I give them the option of structuring True-False or Multiple Choice or Short-Answer Essays, for example, and that often is instructive for them, too, in how tough the task can be in writing an exam.
All of this often yields useful feedback for future exams as well as for teaching the courses the next time around, with the information as to aspects of course content that especially intrigue students. That has sent me down some research paths for more information on those aspects with rewards for me as well.
paul_denny - December 1, 2010 at 2:37 am
For several years, I have regularly challenged students to author and share assessment questions. Although I have not always used the created questions in formal exams (which is the main focus of this article), I agree with earlier comments that the process itself presents many useful learning opportunities for students.
For those who have been conducting this activity manually, you may be interested in a free web-based tool called PeerWise that automates the process and is very simple to use. As students contribute questions they also include explanations of the correct answer in their own words. They can earn achievement badges and points which (for some students) can be useful motivators. Importantly, PeerWise allows students to evaluate the questions they answer by way of ratings and comments, and the shared ratings can be used to help students find good quality questions in the repository. Students can also subscribe to authors who are writing questions that they find most useful. Although activity on PeerWise is anonymous to students, you can keep track of participation on a per-student basis.
If you are interested in finding out more, have a look at the PeerWise site:
http://peerwise.cs.auckland.ac.nz/
anna_smith - December 1, 2010 at 9:08 am
This year I had students in an undergraduate course write their own midterm. They wrote a question for each major concept, and provided acceptable answers for each question. Then for each Q and A, they described how they decided the question was important and the answer was acceptable. For this last part, students quoted readings from the course, content I presented, and described small group conversations from class. I found this last part particularly helpful for me. I could see what students remembered, which resources they were drawing from and what had made an impact.
Students were overwhelmingly positive about the write-your-own-midterm experience. Several of them said they studied harder and felt that they knew more for this midterm than any other. One guy said he actually had fun writing the midterm, and felt satisfied with his work, which he had not yet experienced in college.