Colleges thought they were enhancing efficiency when they moved their course evaluations online, but an unintended consequence of the shift to evaluations not filled out in class is that students started skipping them altogether, The Boston Globe reported today. According to the Globe, some institutions concerned about the dropoff in participation are offering incentives to students, such as lotteries for iPods or meal vouchers at Northeastern University and pizza for the class with the highest return rate in some MIT departments. The Globe says other institutions are considering withholding students’ grades until they submit the evaluations.
- 1255 Twenty-Third St, N.W.
- Washington, D.C. 20037
- Copyright © 2013 The Chronicle of Higher Education
This is so true. I had such an abysmal return on the only course I’ve taught with online reviews.
I can assure you, withholding the grade is the only assured method for obtaining a response. However, that might backfire on the professor. I can relate to this sort of tiresome online request because every time I get one from a company (including requests for any kind of technical support) I now delete it. Very tiring and repetitious.
This is a phenomenon we, too, have observed – much to the dismay of our faculty!
I believe that Yale withholds a student grades for some length of time (2 weeks?) beyond the normal grade release date of those students who complete their on-line evals.I have been told that they, Yale, have a 90%+ response rate for their on-line evals…..any Yalies out there to verify this??
Student evaluations are pretty much worthless anyway, so what’s the difference?
Not good news for us as our faculty would like to have a more reliable and less cumbersome instrument for students to evaluate course instruction. We currently use the SIR II instrument and when coupled with the costs involved in administering the paper and pencil version is cost prohibitive for us in these economic times. Now to learn that response rates are dwindling for online instruments–SIR II online version is still cost prohibitive for us–and we must reevaluate our current move to an online University specific instrument as it seems futile. Maybe we need to think outside the box about this issue. How valuable are the current processes now in place. What alternative solutions would be more meaningful? Should we be Twittering? Facebooking? Exceptions reporting? That is, “If your professor is outstanding (sing their praises) or sadly under-performing (vent and pummel them) please complete this form? Otherwise it is considered that your professor is performing satisfactorily in the classroom in all areas of concern.” Obviously, I don’t have any answers but I would like to hear what others think. Has anyone asked their students what they think about the value of evaluating course instruction?
I do not think there is a viable alternative to filling them out in class. I know it costs money, but if we are serious about using them for improving teaching or making tenure and promotion decisions, then we better our money where the mouth is. Administration is always concerned about the cost side of it. In our school, online evaluations resulted in more than 50% drop in student response rate, while in class you could get close to 90% and above. Now how do you legally make a case for denying promotion or raise on the basis of 40% student response rate. Moreover, with online every student is not filling out the evaluation at the same time.
Our student evaluations have been quite consistent between and among courses. We embed them within our online courses as a normal course activity and have return rates in the 80-88% every quarter throughout the past few years. Our return rates for our on-campus Noel-Levitz paper & pencil surveys is usually below 30%.
The students at our institution have expressed concern that any online evaluation process that requires authentication cannot also ensure their privacy and anonymity. We have seen a trend toward mandatory online evaluations with all responses coded either average or excellent. Is there any value to this?
Fyi
If there was ever an indictment of the validity of this charade, this is it. Schools use these “evaluations” (satisfaction surveys?) for tenure and promotion purposes yet the students think so little of them that they won’t bother to fill them out unless: 1) its all but done for them by allowing them to do so in class (usually at the end of the semester when angst levels are high), or 2) they are being bribed to do so, or 3) they are being coerced! And yet the results of these adolescent popularity contests are still used to determine the professional fate and future of a Ph.D. who has spent half their lifetimes preparing for his or her profession! Truly amazing! All of us in our day to day interactions must constantly consider the impact and consequences of what we say and do in public in terms of the possible repercussions; it’s an everyday aspect of life. Yet inexperienced teenagers, many with an ax to grind, get to rate their professors anonymously with no requirement whatsoever for objective proof to back up their impressions. And then the results are somehow considered to be proof of effective teaching or the lack thereof. And this is how some of the most educated people in society choose to validate what they do?
One of my friends at a northeast liberal arts college has an interesting solution that I may implement in my classes. She has YY students in her classes and gives the students private class time on the last day of class to fill out the online evaluations. A week later, on the morning of the final, she checks with the department admin for the metadata. If she don’t have YY evaluations registered by the morning of the final, then no final for anyone. With the final worth 30% of their overall grade, they can go from an A to a C – or worse – in a heartbeat. She reports no problems with evals not being filled out.
Many of the responses to this point are predictable: a) handwringing about a limited report in a newspaper without much information about what’s really happening or how extensive it is; b) ire from people who believe evaluations aren’t valuable, or that they’re “adolescent popularity contests,” and inherently invalid; c) a few notes countering the doom and gloom.Yet all of us are researchers, presumably. Read the research! There’s likely to be dropoff on online evaluations if a) there’s no follow-up, or reminder to students; b) there’s no understanding among students about why these matter for improvement of the course or evaluation of the instructor; c) the instructor doesn’t set a context for them, or contrarily dismisses them as unimportant; d) the institutional climate disempowers student voices. Our institution went from fully on paper to mostly online within two administration cycles of surveys–faculty found that response rates stayed high due to multiple reminders both from the system and from a staff assistant; customized questions added by instructors targeted specific class issues to which students could respond, including open-ended text boxes for comments; faculty talked about how they had used and would use the evaluations. Where these things happened, at most we’ve seen a 15% decrease in responses–and usually much less–staying up in the 75-85% response rate, almost exactly what we had when they were done in class. Where these things don’t happen–particularly when the faculty member fails to mention them or publicly disses them–response rates fall to 50% or so. Faculty also found a minimal shift in responses, in terms of affirmation or criticism, and found that student comments increased significantly; most instructors said that the comments were better focused and proved more helpful than when they were on paper. And turnaround time went from about a month to around two weeks.
I am so sick of the whining. Yes, students have an absolute right to judge their professors. If people do not want poor evaluations, then perhaps they should sped some time learning how to teach. And if you don’t like being evaluated by teenagers, QUIT YOUR JOB.And if the questions on the evaluation form seem like an “adolescent popularity contest” how about committing some service time to working on the committee responsible for redrafting the questions: there is such a committee, you know, and YOU should be serving on it rather than complaining about the unfairness of the universe.
Amnirov–Perhaps you should listen to some of the criticisms of teacher evaluations. Did it ever occur to you that different places have different cultures? I have had issues with the process myself — not that my numbers are bad, per se, but certainly not as good as they should be. How do I know? Because I keep my office hours. I don’t miss them. I’m not late for them. That is something that is quantifiable and should not be open to interpretation. Yet every semester, I have a handful of angry students who “F” me (no profanity intended) all the way down the grading sheet because they are angry that they are doing poorly in the class. I teach writing. Some people are poor writers, and don’t like it when I don’t “praise them for their creativity.” And as for your suggestion that I serve on the committee, I am not tenured and therefore, am subject to their whims.
Yes, my institution has also seen a significant decline in the percentage of students completing online evaluations. It appears that students who do complete an evaluation do so either because the student loved the course or hated it. The vast middle has disappeared. You might be able to withhold student grades for a while but only for a while. This has not seemed to make a difference at my institution. Faculty, in general, would much prefer to go back to the paper-and-pencil evaluations completed in class. We received many more comments and more useful comments. Now we get very few comments, and the comments made are of little value (“loved this course” and “worst course ever” type comments).
Whether being evaluated by college students, supervisor, or participants in a workshop there will always be a need to sift the information for useful tidbits. Why all the moaning? In a previous college where I was employed we switched to online instructor evaluations with not the slightest hiccup. Why? Because we did as a previous poster suggested. We made several announcements in each class, we were careful to ensure the students’ anonymity (signature was optional, with many provided), and we made a point to indicate how the evaluations would be used to make class / instructional changes. One benefit of the online evaluation was that students could do the evaluation on there own time schedule within a two week window. That flexibility remove some of the angst noted in “evaluation of the moment” when required in a specific class period. As with any data, a variety of perspectives must be used to tease out useful information. Furthermore, a single evaluation event is typically worthless, must like a single data point in a lab experiment. Repetition of results will allow trends with some validity to be obtained.
I am hopeful that — with the arrival of devices such as the iPad — that we will be able to take advantage of the benefits of electronic/online evaluations, while simultaneously going back to asking students to complete this process in class. When they are asked to do it in class, and are given specific time to do it, they usually do it (many don’t understand that it is an option).In the meantime, why not use cell phones to text their comments to a common system?We’ll get there.
The students at the Illinois Institute of Technology have been fighting for professor and course reviews to be published online for most of a decade now. When the system was revised last year, participation rose from 20% to a more meaningful 60%. However, the results are not yet published online, almost a year later. The Student Government Association is still in negotiations with the Provost to get this done. The professors, we are given to understand, do not want the ratings to be made public. The revisions to the system took the number of questions down from 20/class to 3/class: rate professor teaching quality, rate class quality, and a comment box. This was combined with a heavy publicity program launched by the university but primarily implemented by the student government via facebook and word of mouth. Incentives for university-wide surveys in the class have included “parties” that have a free pizza lunch after a student has filled out the survey on a provided laptop.
Course evaluations absolutely have to be public.
Seems like the same problem of getting citizens to vote in the general elections. Doesn’t it?When you have to force, or bribe, students to do something, haven’t you crossed onto the slippery slope of ethical behavior?Historically, on our campuses, either the very satified or the very unsatified fill out a survey. The ‘silent majority’ say nothing.Let’s be honest. We do this to satify accreditors.And, all they want is paper trails.
If I may be allowed, three further comments: (with apologies for being slightly off topic)Course evaluations are in fact instructor evaluations. Students by virtue of the fact that they are just that, are- for the most part – not capable of judging the suitability of course content or organization, or the extent to which the professor knows the material, if they were they’d be graduates. And so many of the questions on these survey forms involve soliciting an opinion about the professor’s behavior in class – which, by implication is supposed to provide a measure of what the students have learned. And this is supposed to make sense?As for reading the research, the “research” on this topic contains some thorough large sample size studies that conclude that “course evaluations” are all but meaningless – other than as a measure of subjective student satisfaction. That being said, there are all too many examples of educational fads that have come and gone, mostly in grade schools, that although supported by lots of educational “research” at the time, proved to be an abject failure (new math, whole language reading, etc.) If scientific and technological progress was based on the same sort of “research” put forth by many educators we’d probably still be living in caves, worshiping the spirits and consulting entrails to divine the future. Lastly, the overarching assumption that a satisfied student is an educated student simply does not hold water. Ask anyone who has suffered through a military boot camp if they learned the required material and what sort of warm and fuzzy feelings they had toward the drill instructors who taught them and the disconnect becomes obvious. People can and do learn perfectly well ( perhaps even more completely) in a “grab them by the – - – - – and their hearts and minds will follow” setting. Not the most pleasant atmosphere to learn in, but one that evidently has proven to be effective in an arena where failure to learn can cost you or your buddies their life.
When I worked at an institution with online evaluations, I simply took my students to a computer lab during class and had them fill out the evaluations. I consistently had a 90-100% return rate on my evaluations. It’s not that difficult.
Given the popularity of sites such as RateMyProfessor, PickAProf, etc., students do want to see ratings of their instructors. I imagine that if students were told that the ratings would be made public if 90% of the students in a class completed the online evaluation, more students would fill them out.
At a previous institution where I worked, all evaluations were done online and in class. Many of the classes had computers, and the ones that were not taught in computer rooms where scheduled for 15 minutes in a computer room. Proctors would get the classes, take them to the room, and let them go after they finished the survey. It took some logistical support, but was very effective in getting the response rates above 80%.
I was looking for some out of the box ideas when I posted my earlier comment: I like the text message and iPad ideas. As for putting your money where your mouth is: personally I would like to use the dollars say, for academic support items; especially if this process is inane. In these economic times our small school needs to be even more judicious than ever in allocating resources. On our campus our faculty and staff are our colleagues and as one of the administrative staff my role, like my peers in this community, is to provide the best mutually satisfying work experience possible–within the boundaries set by authority–while all of us strive to ensure the continuous improvement of effective learning, in and out of the classroom setting, for our students.Several of us will be meeting next week to discuss the use of a newly designed online course evaluation instrument and I wanted to be better informed as to what others in our field are doing and thinking about this process. I wondered if assessing course evaluations for tenure decisions has out-lived its useulness? Do course evaluations provide us meaningful data and information? Are there instruments being used by others that are considered highly meaningful? Are there more effective alternatives to course evaluations? Can course evaluations be used to improve teaching and positively effect learning outcomes and not be used to decide tenure questions? What alternatives are there to course evaluations to evaluate an instructor/professor’s effectiveness in transferring knowledge to the student?What are the goals of course evaluations? What should be the goals for course evaluations? How best do we meet the intended goals?
I have observed the same phenomenon! Non-mandatory course evaluations are normally only completed by students who view their instructors between two extremes — atrocious and marvellous.
The ability to increase your response rates with online surveys is, in my opinion, directly related to how relevant the student feels their participation will be to the policies, procedures, academic quality and student services they interface with during their time at the institution. If, for instance, they feel that the library services are lacking and these services are never enhanced then they are less likely to respond in the future. The consensus will be that nobody is listening. A previous poster stated that students are unqualified to judge faculty. That may be the case in amny situations but there are many areas where their opinion does matter so why not start with that. If students don’t like the library services and complain about it then do something to improve it AND THEN communicate like mad that “they spoke, you heard and you made changes”. If students can identify that their voicing of their opinions actually resulted in improvements they are more likely to be willing to continue voicing their opinions. As per the students opinions on matters academic – one hopes that school administrators and faculty committees are wise and mature enough to discern the wheat from the chaff as they review the student responses. Individual faculty should be open and accepting of what their students have to say as there may well be something in these responses that help make them or push them to be better educators. The secret: build an awareness around knowing what the surveys are for, how important they are to the institution’s future and the student’s academic experience and how they can result in real and meaningful changes.
I agree with the constructive comments made by Bethcollege. We utilize an online survey and initially, yes, the response rates were lower than face-to-face, pencil/paper surveys. However, our Academic Senate decided to be proactive and address some of the issues mentioned on the blog. We developed a list of tips to encourage student participation, most of which are mentioned on the blog, except for the withholding of final grades due to noncompletion. We provide a list of computer labs available during each semester where faculty can take their classes, if they choose, on a “survey” fieldtrip. We encourage faculty in write-ups within their tenure, promotion and contract renewal documents to discuss the survey results as only one part of teaching effectiveness. The costs of administering the paper/pencil surveys, the lack of timely results from paper/pencil surveys, plus the increase in the number of online classes we offer required us to convert to online. Yes, change is hard but to achieve buy-in from faculty, our Senate Executive Committee led the initiative to transition to online and worked with our administration to consider alternatives and the process for making the change.
I would not want to use grade withholding to strongarm students into filling out the response. The end of the semeter is filled with so much angst and anxiety as it is. Adding force to fill out the form could backfire on us very folks who count on pointed feedback. I rarely pay attention to the quantitative information. I find the comments most valuable. Sometimes the comments sting, but usually this information has merit to some degree. I need to know what was missing, done well, or needs changing and it seems like the only way for me to get this information is if students feel they have agency and they are not rushed into responding. We have the option of paper or online. I opted for online on two occassions. All I received was numerical ratings, but I didn’t get any specific teaching feedback. I stick with paper forms and I distribute them at the beginning of class so that the students can write comments.
A grade should not be withheld pending an evaluation. Unless this is written in the student handbook, I can foresee successful lawsuits against this practice. That said, I did get a much greater response rate in paper evaluations than I have with online evaluations.
Most students have a laptop, an iPhone, a Blackberry, or an iPod Touch. Instead of suppressing these media consumption devices, leverage them. Have students break out their cell phones or laptops and do the evaluation in class.
Our students have the impression that evals don’t make much difference. They’re right in that we really only look at the mean score on the “effectiveness” question. And then we expect everyone to be above average.Response rate is low and students say they worry about confidentiality. I don’t remember hearing this concern expressed when the surveys were done on paper, in class.
#5 above: “Student evaluations are pretty much worthless anyway, so what’s the difference?”Well, that’s honest and to the point–and probably true.
Why should students bother to comment? What good does it do them in the short run? Try making the results public for the students. Then perhaps they will value the input of their peers and contribute to the project.
Student evaluations are one of the disasters perpetrated on higher education in the ’60s when the inmates (students) were allowed to run the asylum, and no one–faculty, administrators, legislators, or law enforcement officials–did anything to curtail the anarchy. 60s students, who were far more adolescent than today’s were often in college to avoid the draft and who were only concerned for the face in the mirror, were always talking about the need for “relevant” courses, and every responsible adult who should have held their ground instead capitulated thus allowing teacher evaluations to be implemented. What obviously should have been happening during this period, if we had listened to many in the “greatest generation” who sacrificed through World War II and Korea, was there should have been massive arrests, expulsions, etc. of the self-centered, me-first generation that have made “what’s in it for me?” replace “how can I help my ____________ (neighbor, country, friend, profession, etc.)?” A problem we still face today–consider Wall Street, politicians, large corporations, etc.It is not just a question of the return rate, but also the exercise’s validity. Does the returned evaluation show a constructive evaluation of the course? Does it show an overly positive or negative view of the instructor? Does it appear to be a course evaluation or a teacher evaluation? Do the written comments indicate a thoughtful consideration, vindictiveness, or excessive adulation? There simply is not enough time to do the research necessary to answer and weight these questions for every evaluation submitted.Finally, there is questionable evidence that students know how to fill out evaluations. They vote for 1 being the best instead of 5 without reading directions. They have not been out actively involved in the field to the point they can be knowledgeable about what is important or not. In short, they are not qualified to evaluate. With the passage of time, one of the “poorest” teachers, at the time, I ever had is the one I quote and refer to most often.The only thing to do with student evaluations of courses and teachers is dispense with them.
It is a pleasant surprise to finally see some benefits accrue to technology in the classroom, namely, the dawning of the long-latent reality that true student opinion evidently is that canned evaluations long served up to student consumers have been overestimated by administrations and faculties as well as the students themselves. They aren’t even worth the time it takes to fill them out. Not to mention the parallel emerging–and more telling–reality that institutions are responding by both bribing students with more idiotic gadgets as well as threatening students to either fill surveys out or have their grades withheld. Student opinion surveys had their time during the VietNam era of the student draft when they originated. Since then, however, they have done nothing but increasingly drive down the quality of education nationwide, with faculties and their students playing a game of “you scratch my back and I’ll scratch yours.” Ethics in the classroom and institution-wide has taken a back seat to what education could mean were it founded first on ethical practice in the classroom.
I have been a faculty member at 3 universities over the last 30 years and I would make the following observations about student evaluations.First, they have value of kept in context. They should never be a major part of any performance review. They should only be used along with other tools, such as classroom obserations. However, it should be noted that most faculty members who complaign about the use of student evalutions refuse to be part of or permit the use of any other method of evaluating their teaching, claiming that everything violates their academic freedom.Second, while it is true that students are not experienced or knowledgable about the subnject of a course or the craft of teaching, they are the only ones who must sit through every class, complete every assignment, and receive feedback from the professor. The rest of us do not have that much experience with our colleague’s teaching.Third, student evaluations can be valuable tools when read properly. Look for trends and repeated comments and you will learn something about your teaching.
Having a faculty member devote time to an evaluation students turn in anonymously and one student collects and puts in the instructor’s mailbox can be valuable as the faculty member alone looks over the comments for consistent reactions; however, other than that purpose they are worthless. When a faculty member on their own solicits an evaluation the quality of responses improves significantly because students see the faculty member seeking to improve. If students with an axe to grind, usually without reason want to rant, they will probably be a tad more realistic if the only person to read it is the faculty member which serves the purpose for everyone better than dealing with vitriol. The student who does it in a hurry without reading directions will have their “reversed” markings (1 for 5) contradicted by their comments thus allowing some value to the evaluation. Given the often political nature on a college campus where colleagues are suppose to be professional and supportive but are often threatened by a colleague who appears to “achieve” more success, this eliminates blowing an excessively negative comment out of context to alleviate the resentment toward “more successful” junior colleagues while still providing feedback. Students having preconceived ideas but lacking experience in the subject matter will have their prejudices exposed. dboyles had it right when he pointed out that teacher evaluations “increasingly drive down the quality of education nationwide.” In this day and age when many students, following their public school experiences that have little or no academic rigor arrive on campus expecting the same, confronting them with challenging work often leads to resentment. Further, if students are so brilliant that they know what should or should not be part of class content, procedures, assignments, and grading, why do we even have higher education at all?
What’s the point to the professor evaluation anyway? No professor will change his or her ways anyway and the administration does not act on evaluation trends anyway, positive or negative. What’s the point?
The comments above a display a widespread lack of understanding of more than 50 years of intensive examination of student ratings and faculty evaluation. The weight of evidence is clear and it tells us that ratings can be reliable, valid, and useful. However, poor practice and misinterpretation and misuse of ratings data have created many problems. The complaints we hear are often a result of these errors, but their targets (ratings and student opinions) are mistaken. There are guidelines for good practice, but as long as we let mythology and unsubstantiated opinion hold sway, we will never realize the possible benefits of good evaluation and its combination with other important systems (like sound instructional design, accurate assessment, and the blending of information from multiple sources). PS: “marchman” needs to retire if his ideas abouot students are still predicated on a view of the 1960s as the demise of higher education and if he thinks throwing out student opinions will accomplish anything of value.