Online Courses Should Always Include Proctored Finals, Economist Warns

May 10, 2011, 4:40 pm

Online economics students do not absorb much material from homework and chapter tests during the semester—perhaps because they expect to be able to cheat their way through the final exam. That is the lesson from a study that Cheryl J. Wachenheim, an associate professor of agribusiness and applied economics at North Dakota State University, will present in July at the annual meeting of the Agricultural and Applied Economics Association.

Ms. Wachenheim is no enemy of distance education. As The Chronicle reported in 2009, she continued to teach her online courses even during a National Guard deployment to Iraq. But she has noticed that her online students perform much worse than their classroom-taught counterparts when they are required to take a proctored, closed-book exam at the end of the semester.

In her study—a previous version of which appeared in the Review of Agricultural Economics—Ms. Wachenheim looked at the performance of students in six sections of introductory-economics courses at North Dakota State. In online sections whose final exam was unproctored and open book, students’ exam grades were roughly the same as those of classroom-based students who took proctored, closed-book finals. But online sections that were asked to take proctored, closed-book final exams performed at least 15 points worse on a 100-point scale.

Ms. Wachenheim fears that students in those unproctored online sections really weren’t learning much, even though their grades were fine. In self-paced courses, many students appeared to cram most of the homework and chapter exams into the final week of the semester. Few of them bothered to do the ungraded practice problems offered by the online publisher.

Then there is the question of cheating. Ms. Wachenheim’s study did not gather any direct evidence, but she reports anecdotally that students have told her how they work in groups to compile huge caches of the publishers’ test-bank questions. She quotes one student as saying, “We may not learn the material, but we are guaranteed an A.”

Ms. Wachenheim’s findings parallel those of a 2008 study in the Journal of Economic Education. That study found indirect evidence that students cheat on unproctored online tests, because their performance on proctored exams was much more consistent with predictions based on their class ranks and their overall grade-point averages.

Ms. Wachenheim ends her paper with several suggestions for improving learning in online settings. Those include insisting on a proctored final exam and reminding students of that exam “early, often, and broadly, so students are ever-conscious that they will be responsible for the material in an unaided environment.”

This entry was posted in Uncategorized. Bookmark the permalink.

E-mail

http://twitter.com/TamCook43 Tamarah Cook

It sounds like the problem may be with the type of test administered to online students, self-paced (not internet) courses, or perhaps it is the misguided student. I have taken many online courses with and without proctored exams. In my opinion the open-book tests are much more applications oriented and harder than traditional tests. Yes I have my book to assist me, but if I do not understand the concepts that have been taught throughout the class, the book will certainly not save me. All tests I have taken are timed, once the time is out you are locked out of your test. Three and a half hours for a test and I will take the entire time.

Perhaps Ms. Wachenheim should consider administering exams that are based on real-life situations and applying concepts learned throughout the course. Maybe internet courses should have more opportunity for discussion among classmates instead of “independent” learning.

Then of course there are always students who think an open book test will be simple. Let them take one final that makes them use their knowledge instead of filling in blanks – let’s see if they choose not to learn the materials in the future.
http://www.facebook.com/profile.php?id=764514991 Ellen Bremen

Our math department (I’m in Comm Studies) struggled with this dilemma when the college wanted them to offer online classes, but at the same time, our testing center hours were reduced due to lack of funding. And, of course, what does a prof do when the student needs to truly take the course at a distance? I’m the head of the campus Distance Learning Committee. Our suggestion? Why not veer away from standard major multiple-choice
exams and focus more intently on students “showing what they know” so
they have to apply their learning? I worry about online classes that do
not diversify the types of assessment offered to students. Also, the author notes that the students are not doing the practice work if there aren’t points attached. I would say add points to everything and ensure that the student realizes that this formative assessment will help them with the end product. Very insightful and important discussion in this article. Ellen Bremen, M.A. @chattyprof:twitter
11284814

Proctoring exams is consistent with open book exams. For a variety of reasons, I now give closed book exams, but in most of my teaching, I’ve used open book exams for the reasons that you describe. Proctoring still has value in preventing people from exchanging notes and answers during the exam itself.

And yes, I think it’s a problem in on-line courses. I was a student in an on-line course in which one student inadvertently emailed the whole class with her favorable response to a proposal to collude on an exam. Except for the two involved, we all got a big laugh out of it, but I’m glad I wasn’t the professor assessing students’ performance in a situation where I couldn’t be sure who was really doing the work.
amnirov

Perhaps econ profs should be less lazy and write real tests instead of merely selecting multiple choice questions from a pool that they didn’t even have the energy to create.
mbelvadi

I see a confluence of problems with this criticism. An apparently unrelated issue that higher ed faces is the transferability of credits. When it comes to basic courses, like econ 101, students and the academic system as a whole expects that approximately the same material is covered everywhere, so credits can transfer. That means the assessment should be measuring the same set of information learned. At some point, then, the actual test questions should be about the same. That in turns means that it’s plain wrong for criticizing the teacher for not coming up with “original” questions – society wants just the opposite from them.
Don’t get distracted from the real elephant in the corner. The fact is that cheating is already at a high rate in regular classes according to surveys, so it should not come as too much of a surprise (unless you’ve been living in denial about this issue) that a greater opportunity to cheat would result in a greater rate of cheating in the online environment.
mbelvadi

How to proctor a distant student: how about with a webcam? Require the student to position a web cam in such a way that the entire student’s body, not just their face, is visible throughout the exam, and have someone watch them the entire time to make sure they don’t fiddle with cell phone texting etc. You could even record the video so that you could play it back if there’s a question of cheating.
cindydietrich

While I was at UMUC, I had to go to a local testing center. The testing center required I show idenitifcation. The instructors included a note with the exam to let the proctor know what I should bring into the roon. If the exam was closed book/closed notes, the only thing I was permitted in the room was a pencil.
msehphdjd

Others have made similar remarks, but… I think it depends primarily on the material and the structure of the course. My course chapters “roll out” each week so that it is not possible to complete the bulk of material early or late in the course. The chapter quizzes are timed and while a quick student could be flipping through the book while taking it, they would likely a) have trouble finding the items, and b) run out of time. The student who can do that well, probably doesn’t need to do so!

The essay final is designed in a way that requires demonstration of their grasp of the concepts and the ability to apply them. If they haven’t done the work/learning along the way, they’re going to have trouble with the final even if they peek at the book. That is, they might be able to look up “social construction” and refresh their memory, but if they don’t have a good understanding of the concept, they will have trouble writing a good answer to the question. Might they pass? Yes. Are they likely to earn and A or B range grade? No. I don’t disagree that in a course that requires primarily objective testing, there will be problems. But, I do think there are ways to address many of the potential problems, besides proctored exams.
dickiea

I hope there are other suggestions for improving this online course that incorporate recommendations for authentic assessment of student all throughout the course. What is being described here suggests to me that this was a poorly designed course that did not require active student learning elements anywhere throughout the course. Also, I agree with Tamarah below who recommends a very different type of final assessment. It sounds like this may be a final exam that is comprised of mostly if not all multiple choice questions. AND – I would almost guarantee you that students are also finding ways to cheat in the procttored test environment which are yet to be discovered.
11889431

I teach online and I don’t give proctored tests and I can tell you the test is mostly application and students do not do all that well. Proctoring wouldn’t make any difference.
12113573

I agree – online courses should focus on active threaded discussions beginning with thought-provoking questions as well as on active learning assignments that force students to apply major course concepts. Attention paid to following these principles while a course is in development pays off in student learning.

However, in content-laden courses, the challenge is to determine the major course concepts and testing makes sense. If exams are absolutely necessary, the questions should tap into higher level thinking. All the tricks for preventing cheating should also be used, such as random order of questions or randomly changing the forms of each unit’s exams.
jfroh

I have been teaching many different online classes for several years and it is very difficult to stop the online cheating. I have developed the courses over the years to break it down to discussions,(every week or two) short answers,(every week or two) quizzes (every week or two) and usually 3 exams. The quizzes and exams are all timed, they may be able to look up a few things but then they are locked out by the time. The quizzes I even let them retake up to 3 times, but they must get a 60 or above as the minimum to be able to go into the next retake. Then it will record the highest. Their grade is made up of all the parts with the exams being about 30-40% of the total. This does take alot of prep work and constant communications with the students.

I do like the idea of the cam and there are companies out there that can do that. I have looked into them but it is expensive.
lindamorosko

Ms. Wachenheim doesn’t state what other methods of assessment were included in the course. Regardless of delivery modality, the traditional testing method does nothing more than test memory skills. Assimilation and application of learning are not evident in a test. Discussion, reports, papers, group projects, case studies, these all demonstrate learning much more effectively than tests. I recommend using tests to assess whether or not students have read the books and other materials. Other than that, be creative and create interesting learning opportunities for everyone involved. On our campus, for instructors who want to ensure no cheating occurs, we recommend proctored testing. Our school has a proctoring process both on campus and for our online students who are not geographically close to the college.
commserver

I find that there is no difference with students in traditional classrooms. I have many colleagues who use a very finite varity of tests. The students are able to amass the questions because the graded tests are returned.

In these situations the students were able to get higher grades. The professors look great because their students were “high” achievers.

I remembered when I was going for my master’s degree students studying in groups with questions from previous semesters.

It doesn’t matter if the student is retaining or not if the students have access to previous tests.

I have made it a habit to create totally new tests. It is more work for me but I can claim that my students who did well did so because they had to.

An analogy can be made to SAT. There are schools that provide samples of previous tests. You can even get books with practice tests.

In the new paradigm of academic testing, schools often teach to the standardized tests, not to the actual material. I find that many of my students don’t have the academic background. I am therefore have a dilemma: Keep High Standards or Water Down?

It is easy to criticize online education but to me it is the system.
juneparsons

Yes, there are several ways to tackle this problem, some of which require us to think out of the box. To give an example, for some online courses, I have structured evaluation so that students take the exam as many times as they like until they get the score they want. The key here is that exam questions are pulled randomly from a large test bank, so each time a student takes an exam, the questions might be different. Sure, the students can write down the answers from the first time around, but that process makes them pay attention to the answers and the tests then become a learning experience as well as an assessment experience.
commserver

I have taught courses in Information Systems and Business Statistics. In those courses I create a situation that the students are expected to provide answers based on the specific situation. I don’t rely on true/false or multiple choice.

It doesn’t matter if a test is open book or not. If the student doesn’t know the material or even have looked at the textbook then the student doesn’t know where to look. I remember having a student start looking from page 1 and continuing. This student wasn’t able to complete the test in the allocated time.
commserver

The use of web cam sounds great but I have a laptop with built in web cam. It is very hard to use the laptop in such a way that the web cam can should the body in the way suggested. I guess I can get a 2nd web cam but I don’t have enough usb ports. I guess I could buy usb hub.
goodeyes

Students should not be allowed to submit homework and other assignments during the last week of class. These assignments need to be spread out during the term and with deadlines for each. In addition, small graded quizes should be given and timed along with a big final exam. On-Line teaching can’t be taught the same way as classroom teaching. Professors must take the responsibility when on-line learning is poor. There is great ignorance on how to teach on-line and standards should be required.
panacea

There is software available for webcams to be used as proctors . . . but it is expensive.

When you’re proctoring 18-20 students via webcam, that’s hard enough for faculty who may not be technologically inclined. Try a large lecture class, and it becomes impossible.
proctorcam

This is Hawthorne’s Law at its finest – The simple act of studying the performance of workers or students will cause their productivity or test scores to improve. In this instance, this has much to do with the online, un-proctored students’ perception of the stakes of the test and ease of “gaming” the system.

We believe proctoring can be an important aspect of building a successful and credible online program. We consult our distance learning clients to have both on campus resources, potentially test center network relationships established in addition to services like ours (online proctoring via the web) for building an accessible and successful proctoring protocol even if it isn’t adopted by all faculty. Any questions, please feel free to e-mail us info@ proctorcam.com.
panacea

I use online exams in my face to face class. It makes item analysis soooo much easier.

When my department first started using them, we allowed students to take them at home, the “proctor” was there to help trouble shoot computer issues. While I could see the test, I couldn’t see the student during the exam. Everyone took the exam at the same time.

A professor caught students meeting in a coffee shop to take the test together, even though questions and answers were scrambled. It actually didn’t help the cheaters . . . nursing exams are application based and if you’re approaching it like a black and white regurgitate the material class (as nursing students often do) then it won’t matter, you’ll still fail. Which is what happened. So we started monitoring IP addresses . . . but some students had data cards which made it difficult to track actual physical location.

I started using a computer lab to administer the tests simply because I got a class who had no experience with online testing, and they were anxious about computer problems during the test . . . the lab relieved those anxieties. Now I administer every test that way. I’m sure I still have attempts by students to cheat, though I haven’t actually caught anyone, but nothing any different than what would happen with a paper and pencil test.

Bottom line is, cheating will always be a problem. How you set up the test makes a difference. If you’re not tech savvy, online testing outside a controlled lab enviornment might not be the best option
for you. And writing good test items always makes a huge difference, whether you are testing online or paper pencil.
torshi

This wasn’t a single online course. It was three different courses with various combinations of online or classroom instruction, pacing, and testing methods. But the comment that a key problem is self-pacing is right on target.

The paper describes two problems. One is cheating, and the other is learning. Some of the steps an instructor can take to reduce online test cheating (questions from a pool, one-at-a-time, reduce minutes per question) won’t help students learn. The paper is mainly focused on learning. Online students learned the material less well in a self-paced course and performed at a lower level on a proctored final. (Allowing students to take several exams in one week, and allowing take-and-retake attempts at the same set of questions, seem to be particular problems.) Exam questions for all sections for all courses came from publisher testbanks. The main problem seems to be that some course structures enable students to learn the questions and answers well enough to do well on a test before the final, or to do well on the final if they can look up the answer, but not well enough to remember the material for a proctored exam–because they never really learned it.

Wachenheim recommends a proctored final, but most of her recommendations are oriented toward improving online learning rather than cheating reduction alone. She recommends telling online students early and often that the final will be proctored, a final with many application questions, removing the self-paced element (requiring homework or practice problems to be completed instead of encouraging it, having test deadlines throughout the course), and alerting them that the course will require more time commitment than they might expect.

She doesn’t address the issue of using and reusing publisher testbank questions. This is a weak spot. I don’t buy the argument made by a previous poster that only by using publisher testbank questions can course credits be considered transferable. I teach multiple sections of a large survey course every year and continually add new questions to my own testbanks. Everyone who teaches this course teaches the same basic content. Syllabuses and textbooks are very similar. Many of my questions are similar to everyone else’s questions who teaches this material, and to publisher questions–except they are not exactly the same question. The rest are factual questions that students either know, or they don’t. If you want to work from publisher questions, you can use those questions as a starting point but edit them, for example turning a definition question into a concept- or fact-application. In my field, publisher questions are not very good anyway (although maybe that’s not true in econ.). It’s also not necessary, even in large classes, to have entirely multiple-choice exams. On the other hand, I’ve had more problems with cheating in online tests with written-answer questions than with multiple-choice questions.
hacucbi

I’ve thought a lot about this issue and have come to the conclusion that having another person verify identity and watch the test taker is the only approach that will work. A webcam will not work because the student can be seated in front of the webcam, looking as though he is working on the test. Meanwhile, off camera, another person can be looking at the test on a separate monitor and providing feedback, or even answering the questions with the real mouse and keyboard.
commserver

I have a daughter who is going to Williams College. She related an experience from last semester 2nd semester Organic Chemistry class. The professor scheduled each test with 2 different times. There were a group of students who would pick up the test paper but would walk out. The students would then go to the 2nd session and then get 90+ on the test. All the other students would struggle to get 80+.

This is an example of lazy professor who don’t care. It doesn’t matter that the test is proctored.
chemistry_guy

This is what I’ve been saying for years: we need more Proctologists.
contingent

I teach the same humanities class online and face-to-face. In both classes the students write out of class essays rather than take exams, and submit them to me electronically. In the most recent round of grading, plagiarism was significantly more prevalent in the online class than in the face-to-face. The students in the online class who plagiarized were also the ones least likely to participate in (mandatory) online discussion. I’m thinking that there might just be an alienation factor here that is difficult to overcome in online classes, one that makes cheating feel less like an ethical violation and more like a strategic solution to a problem. Never do the students who cheat ask me for help with their work before it’s due, while the others often do. And I am happy to give it. You do everything you can to create a sense of community and personal connection in an online course, but if some students wants to drift along on the margins, they will.
victorl

For basic skills in a discipline you are studying, perhaps, for the first time (i.e., a “101″ class), memory skills are important. Assimilation and application of concepts cannot readily happen if every thought and task involves lengthy (or even brief) looking up and research of basic concepts. Becoming proficient in a subject means internalizing and learning concepts, interpretations, formulae, vocabulary, paradigms, etc. This is neither trivial nor inconsequential to success. It’s merely unfashionable in our era of “everyone can succeed,” and “multiple intelligences.” No, not everyone has an aptitude and ability for complex subjects. And often the introductory concepts and ideas really do have to be memorized if you hope to work creatively with them at a more advanced level.

An article that appeared in the Chronicle a couple of years ago, written by Christopher Ferguson (2009), probably said this most effectively: “Multiple intelligences provides a kind of cover to preserve the fable: ‘OK, little Jimmy may not be a rocket scientist, but he can dance real well. Shouldn’t that count equally in school and life?’ No. The great dancers of the Pleistocene foxtrotted their way into the stomach of a saber-tooth tiger.”

Let the Econ 101 students enjoy (or not) a bit of rigor in their studies. It will be good practice for their further schooling.
bernardjsmith

Seems to me that one possible solution is not to use exams that test memory but to employ continuous assessment that tests understanding and to require that the continuous assessment be based on active discussion in the online class and on the submission of essays that incorporate aspects of the discussions.
Prof_truthteller

Proctoring also guarantees that the student who is taking the test is in fact the same student taking the class since most proctoring services require an legal ID. There is usually a fee, sometimes as high as $40 and travel to the site both of which could be difficult for the students and counter to the benefits of online education.
11272784

Stating that all tests should be proctored is is a ridiculous assertion. In many classes it’s not hard to design assessment so that essays or tests are open-book, open-web with a time constraint, or otherwise structured so that cheating is difficult or irrelevant. Imposing a proctored, closed-book exam on students is often unnecessary. It depends entirely on how you design the class and the assessments.

Note that nowhere does the article state whether the 15-point difference observed (an entirely anecdotal observation) makes any difference! Maybe the students who averaged 15 points lower were perfectly competent in the subject matter but simply scored lower. As Garfield would say, “Big fat, hairy deal.”

LIFE is not closed-book. When you have a professional challenge, you
don’t need to cough up a complete answer on the spur of the moment
without using any references or resources. I much prefer to assess
learning in ways that let the student demonstrate what they DO know,
instead of trying to find out what they DON’T know. Sometimes what they don’t know is of little importance or merely tangential to the main goals of the class.
Guest

The downsides discussed here are all legitimate issues. I recall working through them with Michael Scriven in the late 1980′s.

However, the retreat to a physical assessment model to bolster the virtual classroom is nothing other than convergent thinking, reminiscent of the engineers pronouncing that bumblebees cannot fly.

First, various forms of cheating can and do occur in the physical classroom. Therefore, the baseline against which virtual assessment is properly assessed is not zero cheating; it is some defensible percent of cheating below which returns do not justify efforts.

Second, there are a variety of virtually mediated ways to reduce cheating on old-fashioned assessments (e.g., multiple-choice tests) to a level comparable to that of the physical classroom.

Third, and most important, high stakes testing with low validity instruments (most instruments used at the college level do not meet minimum validity tests), fails to exploit modern learning and measurement sciences. Good assessments (i.e., reliable, valid, useful, predictive of desired proficiencies, etc.) do not provide an easy way to cheat. On such assessments, students may and are encouraged to consult all of the resources they desire because authentic judgments and defenses thereof are being assessed. The ability to recall and state facts and opinions is not being assessed except via subsumption as a necessary condition to an argument, defense, synthesis, etc. (See: Barnard J Smith’s comment below.)

Retreating to physical assessment in a virtual classroom communicates a failure to keep up on multiple levels. Let’s see if we can get out of the 1930′s box.
cajed

I took a traditional, classroom Ph.D. course in microeconomics in a large public university. The mid-term and final were open book; students could bring whatever they wanted to class. It didn’t matter much. Students who did not master the material beforehand were doomed.
http://twitter.com/TamCook43 Tamarah Cook

I agree with you. And in the end – I would rather hire someone who can apply what they know. Why do I care if they can rattle off a bunch of memorized terms if they don’t know how to apply them?
tmart42

As the proctored exam coordinator for the Office of Distance Education at Boston University, I, unsurprisingly, strongly support proctored final exams for online courses.

There have been questions about how best to proctor exams for students that are at a true distance from campus, which the majority of the BU online students are. We have taken a multiple option strategy to address the needs of our students. We use both a testing center network, and a remote proctor. The remote proctor utilizes a webcam and software that is similar to a student giving a WebX presentation.

Both the testing center network we use and our remote proctoring partner help maintain the integrity of our final exams, but for me that is a secondary, albeit important, benefit of proctoring final exams. For me, the primary benefit for an online class/program/institution is identification verification. Both our remote proctor and the testing center network require a valid, government issued, photo ID that matches the student name that I provided to the proctors. Verifying identification helps make sure that the person receiving credit for a course is the person actually taking the course. This helps maintain the integrity of the exam, the course, the program, and ultimately the institution providing the distance/online learning opportunity.

As more students begin to take courses online, I believe that it is imperative that proctored exams become an industry standard. While the stigma against online/distance education is beginning to wane, there are steps we can all take to help hasten the greater acceptance of distance education. Proctored Exams are a good start.
tmart42

Additionally, the search for proctoring solutions need not be as daunting as it was even 5 years ago. There are several companies that provide in-person proctoring solutions, while several more provide remote proctoring using a webcam and various security features.
Unemployed_Northeastern

Using proctoring services will cut into the for-profit’s healthy margins; ergo, it won’t happen. That is all.
Guest

I have no idea what this topic has to do with type of control, especially your remark since either publics or independents as a single category deliver more online education than the few for-profits that do. For-profits are roughly 11% of market. The majority of them are small and have no online programs. For most of the few large schools that deliver a lot of online education, their on-ground programs are still the largest. DeVry & UOP together have more than 200 on-ground campuses. It’s an informed estimate but I would place the proportion of all accredited degree granting online higher education delivered by for-profits at less than 15%. Perhaps someone else has a more precise figure. Again . . . relevance?
http://twitter.com/casey_marks Casey Marks

Like to see folks outside the testing industry looking at this….
jbarman

“Using proctoring services will cut into the for-profit’s healthy margins; ergo, it won’t happen.”

Sorry, Unemployed. I work for a for-profit that administers over 5,000 exams per year. Every single one of them is proctored.
jmorganpui

Around 70+ institutions are already using a service called ProctorU. This makes it a bit easier for students to take exams anywhere, but still gives faculty the integrity they need.

I work with ProctorU, and we have been discussing this issue with schools for quite sometime. It’s always interesting to me that there remains a group of people out there who don’t think cheating is a real issue in distance ed. Some faculty don’t realize that distance education, when done incorrectly, can seriously damage the reputation and academic standing of an institution. When done correctly, it tears down barriers and educates those who previously had no ability to attend classes.

Another poster mentioned something that I hear from some faculty… the idea of rewriting a test so that it is (for lack of a better word) virtually “un-cheatable”. I love this concept, but I haven’t seen it hashed out in a way that can be scaled just yet. Does anyone have any examples of where this has worked?

Either way, if anyone would like some more information about proctoring over the internet, watch a few of our videos on ProctorU.com or reach out to us.
jmorganpui

good for you guys!
betterschool

This entire logic goes to antiquated teaching and learning methods and the sciences that support them. Knowledge, even of particulars, is much more readily generalized to new contexts (this is what we want, right?) when taught and assessed in ways that focus on higher order behaviors that cannot be performed without deep knowledge of the lower order particulars that get assessed on multiple choice items which, if they are good, possess discrimination indices of 0.65 and predictions of competent behavior bordering on zero. It is professionally irresponsible to be teaching behavioral sciences in a way that is ignorant of them.
softwaresecure

As a company focused on enabling institutions to offer distance learning exams with the same level of academic integrity as a traditional exam in a proctored setting, I am pleased to see Professor Wachenheim renew this important discussion. More faculty and administrators must understand that technology can be deployed that cost effectively enables online students to test with integrity, and on their own schedule — just as they now learn. See our additional comments on our blog:
http://www.softwaresecure.com/us/news/blog/11-05-11/Students_Will_Cheat_–_So_Deal_With_It.aspx
missoularedhead

Instead of relying on tests, I have my students summarize both the textbook chapter for the week and primary sources. In the case of the former, I also have them respond to it…what did they find interesting, confirming, surprising, etc. Yes, I give an open-book, multiple choice final at the end of the class. But their grades rely on the chapter responses and other assignments, not tests on which they could easily cheat. Some students, who were looking for that easy pass, aren’t fond of my methods, but I’m not here to make it easy, I’m here to help them learn something.
http://twitter.com/drsolis Dr. Solis

This is a prime example of why more authentic assessments, perhaps a project-based learning approach, are needed for online courses. Traditional assessments for a final exam should be thrown out the window. Instead, a more practical approach would have students work on a project that has real-world implications and be assessed from that. Electronic portfolios is another example.
betterschool

Nice to see MissoulaRedHead.

One of the modern principles of learning and evaluation you are drawing upon has to do with the the relations between validity and the number and kind of measures. By taking multiple low stakes measures over time and in different ways, even if only two, you are significantly increasing not only the validity of the assessment but the chances that students will retain and generalize the knowledge. By asking students to engage critically with text content you are also increasing the kinds of skills we profess to be teaching and which multiple-choice tests neither teach nor assess very well. You might also consider varying the methods of engagement toward greater authenticity; e.g., consulting multiple resources to solve an authentic problem that will be assessed via a rubric that students have reviewed in advance. The rubric should address form and content and should examine content at levels of knowledge, analysis, synthesis, quality of solution as well as rationale offered for the solution. Feel free to weigh various dimensions liberally in accordance with your goals.

There are a few contexts where multiple-choice tests, assuming they are subjected to validation, are appropriate . . . but very few. One example is when you need to know if someone knows applicable laws, rules, or policies. For different scientific reasons, most uses of either the multiple choice test or the essay test in higher education are professionally irresponsible. The multiple-choice test in particular vitiates the right of an instructor to claim he is concerned that students develop critical thinking skills. There is one kind of standardized “choice” test that taps critical and other higher levels of thinking. It is the the multiple-ranking test that some of us developed quite awhile ago. The items are so difficult to develop and validate — also to score — that I don’t recommend it for classroom use.
arrive2__net

I think its a good idea to proctor final exams online anyway.

The research under the report link is more complicated than it seems at first glance. The students self-selected to the online or classroom conditions, and there were more differences in how the treatment groups were treated than just ‘online or classroom’.

The anecdote about cheating with “caches of the publisher’s test-bank questions” is unlikely to be only an online phenomenon, but is likely to occur in classroom courses as well where the opportunity is presented. When I was in college there were legends that most fraternities and many dorms had file cabinets full of tests and test items.

Since Econ 105 students were allowed to retake chapter exams an unlimited numbber of times, it would have been useful to see a comparison of how many times the classroom and the online students retook the exams.

Bart Schuster
Arrive2.net
Twitter.com/arrive2_net
dmelia

1. Duh 2. Skype
annon1234

THANK YOU!! One of my big complaints against the online school I teach for is that you can get through an entire MBA program and NEVER TAKE ONE EXAM! EVER!!! And for the undergrad students – well they have quizzes – open book open note, test bank questions where they have one hour for every 10 questions (not to mention that many test banks are now available online for a price). That is waaayyyyyy too much time for 10 questions. I am more used to the 40 questions in 50 minutes closed note close book and proctored model. I was floored to discover that there were no proctored exams. When I taught some online classes for a state school there was a midterm and final that had to be proctored.
missoularedhead

oh, there are other modalities I use in my online classes, such as wikis, discussion boards, and video and image analysis (amazing what you can get out of students watching a 3 minute cartoon clip and being asked to analyze it!). And I still use more standard methods (short answer, essay, maps) in my face to face class tests. I completely understand someone adjuncting 10 classes in 3 schools — and yes, I know people like this — using online, self-grading tests. Heck, it would make my life a lot easier if I did that! But I do the final as a multiple choice test to give students who feel that they aren’t learning anything without taking a ‘test’ some validation (largely because no matter what I tell them, they don’t think the other methods are valid…hello, NCLB!)
betterschool

Understood.

The erroneous belief that multiple-choice and similar tests are the only way or even a valid way to assess the kind of learning that we purport to deliver will die slowly and, as we can see in our classrooms, not just in the minds of faculties. I can recall students who didn’t feel that they got their “money’s worth” unless they were subjected to a long, difficult MC test. Notice how many posts here fail to recognize the fact that the tests they are speaking about are statistically likely to contain 20% or more items possessing negative discrimination indices (good students miss the questions; weak students get them right) or have only two robust possible answers out of five (making the item a true/false question with 50% chance) . . . and so on. And this does not even address the mountains of evidence that even the most valid MC tests tend to correlate poorly with competence, and never correlate as well as authentic assessments for which cheating is a difficult or useless endeavor.

Here is one thought that the professoriate at large has yet to grasp. Given that: (a) grades and degrees can determine jobs, income, graduate school admissions, scholarships, and other material benefits, (b) many grading practices can be proven (presumably in court) to be invalid, (c) methods for valid assessment are extant in the scientific and professional literature and can be learned therefrom, (d) it is arguably the professional responsibility of someone accepting payment to teach and assign grades to to do so in a way that is valid with respect to current knowledge (including within and among students because sometimes a benefit goes to the highest grade(s) in a group), it follows that a student who has been materially harmed by invalid grading could prevail in a court of law and be awarded damages. So far as I know, this has not ever happened but, unfortunately, there are those among us who would like nothing more than to press such a claim. I hope it never comes to this but I can’t imagine the day being too far off. Thus, another good reason to grade well: fairness and equity.

In the meantime . . . we argue about proctored tests.

I enjoyed the talk.
craigc

Why not have all tests open book? What is the point of cramming for tests? Memorization of facts is not something that one needs to do in the real world.
arrive2__net

I think it’s a good idea to proctor final exams online anyway.

The research paper, under the report link, is more complicated than it seems at first glance. The students self-selected to the online or classroom conditions, and there were more differences in how the treatment groups were treated than just ‘online or classroom’.

The anecdote about cheating with “caches of the publisher’s test-bank questions” is unlikely to be only an online phenomenon, but is likely to occur in classroom courses as well where the opportunity is presented. When I was in college there were legends that most fraternities and many dorms had file cabinets full of tests and test items.

Bart Schuster
Arrive2.net
Twitter.com/arrive2_net
nugatory

How did the Greeks prevent cheating on phlogiston tests?

Are the cheating police paying attention to what is being said about the relevance and usefulness of this kind of assessment?
marka

Glad to see that a few folks addressed a more serious problem: having someone else take a test for you. In some states, the Bar Exam for attorneys has had to have fingerprints taken, because … some applicants want to cheat, including the Ethics Section of the test!!! So … proctors, yes, as well as some positive way to ID who is actually taking the exam.
drjeff

There’s a big difference between using a question from the publisher’s test bank and using a question LIKE those in the test bank. When I was teaching, I might use test bank questions for quizzes, but rarely for tests and never for finals. I find the questions are just not good enough, generally. Last year, my daughter (in high school) had one teacher who was not returning, who used a final entirely from the test bank. So many of the questions were ambiguous, poorly worded, not so relevant to the material, or bad in some other way, that there was nearly a riot when the proctor, who taught an unrelated subject, was unable to help.

So, no: I’m not a fan of test banks. And I don’t think anyone who pays attention and cares about assessment should be, with their current state.

On the other hand, I AM a fan of software that lets you create many different tests (and answer sheets) from your own questions, by re-arranging the questions. I had one class where I thought cheating was a problem, where I made 6 versions of a test, so no-one would be within sight of another student with the same version. The students actually had the nerve to ASK me how many versions of the test there were! (And, yes, some students had a big drop in their scores for that test.) I noticed from the class participation that the students were studying more after that.
drjeff

No, the great dancers were able to avoid the saber-tooth tiger. The great economists and philosophers, on the other hand…

Thankfully (for those of us who work sitting down), the modern environment is different.
mbelvadi

I think you’re assuming that cheating is difficult in those situations assuming that the person who is going to get credit for the course is the person who has been following the course all semester AND completes the assessment. But if no human being is EVER required to present themselves with photo ID, how could you ever know that Person A didn’t just pay Person B to take the entire online course for them?

That said, I agree about life not being closed book – I get along fine in my professional life having to occasionally consult professional manuals that others might have just memorized.