Students give the same responses on paper as on online course evaluations but are less likely to respond to online surveys, according to a recent study.
The study was conducted at Kansas State University’s IDEA Center, a nonprofit group that tries to improve how colleges use course evaluations. It examined data that the center collected from classes at nearly 300 institutions between 2002 and 2008, of which 89.9 percent used paper surveys and 10.1 percent posted surveys online. The study analyzed student ratings data from 271,727 classes that used paper surveys and 13,101 classes that used online surveys.
The only meaningful difference between student ratings completed online and on paper was that students who took online surveys gave their professors higher ratings for using educational technology to promote learning.
Seventy-eight percent of students enrolled in classes with paper surveys responded to them, but only 53 percent of students enrolled in classes with online surveys responded. The low response rate for online surveys could be a disadvantage, explained Steve Benton, a senior research officer at the IDEA Center and one of the paper’s authors.
“If you have lower response rates, you’re less inclined to make summative decisions about a faculty member’s performance,” Mr. Benton said.
However, Mr. Benton said, a faculty member could still use a survey with a low response rate to look for ideas for improvement.
Because online surveys offer so many advantages—-they save class time, eliminate paper use, and make data entry easier—the paper says that instructors should learn how to encourage higher student response rates, and the IDEA Center has a list of ideas for doing so on its Web site.
While the majority of instructors still administer paper surveys, the number using online surveys increased from 1.08 percent in 2002 to 23.23 percent in 2008.
“There will always be some who stay with the paper survey,” Mr. Benton said, but “it will go up, particularly as we become more environmentally conscious.”
13 Responses to Online Evaluations Show Same Results, Lower Response Rate
22257527 - May 6, 2010 at 5:08 pm
At our university all course evaluations are online and we have a very high respons rate because we tie it to a ‘bonus’. Students who complete online evaluations for all their classes get early access to their final grades.
inanna40 - May 6, 2010 at 5:48 pm
We also offer early access to grades and our response rate went up to almost 90%. Other advantages are that instructors reports that students give more thoughtful open-ended responses with online evaluations.
laurencejgillis - May 6, 2010 at 6:08 pm
Early access to grades, to induce participation in course surveys: I like it ! That is certainly better than any other inducements. And, the higher the response rate, the more credible the “database” from which conclusions may properly be drawn.Thoughtful students with useful survey comments that improve the value of a particular course are a blessing, for which we should be (and are) thankful.What remains unaddressed in this article, however, is the potential for abuse or misuse of student evals in assessing the performance of the instructors. After all, we all know that academic managers have a genetic predisposition toward keeping the “customers” happy, and that should be guarded against.Academe is not (or should not be) a consumer industry. Specifically, we should not have to serve at the whim of students with agendas or grudges. Our obligation is to deliver a quality product. (In its own way, this is somewhat like electing judges, if you know what I mean). I do not knowngly pander to the galleries, but the temptation is there, as long as student surveys are statistically unreliable or if the results can be abused or misused by educrats.
chemmilt - May 7, 2010 at 5:22 am
Putting these online enables the old saw, “Bad news travels fast.”
sorryafk - May 7, 2010 at 9:07 am
Does the research differentiate between online, hybrid, and face-to-face classes? Are the results broken out based on each? Were the same questions asked of a face-to-face class that were asked of an online class?Doesn’t the large percentage of paper survey (89.9%) versus the low number of online surveys (almost 10.1%) has a statistical impact on the results?
boiler - May 7, 2010 at 9:48 am
The online evals do have some advantages, but I was disturbed by the line in the article that “If you have lower response rates, you’re less inclined to make summative decisions about a faculty member’s performance.” Properly used, evaluations are a way for instructors to get feedback about teaching and improve the structure of their courses. There are real problems with administrators using them to judge “faculty performance.” Yet as they go online, evaluations are increasingly going to become the easiest and most official-looking place for administrators to get information about faculty teaching, and that means that they’re going to become more and more dominant in decisionmaking. It sounds innocuous enough to say that you’re making this data more accessible. But unless you make other data equally accessible — and I can’t imagine any way to do that — online course evaluations are going to become the only way that faculty teaching performance gets evaluated. A little example from my own experience. For years, I distributed paper surveys to my classes asking for fairly detailed comments about the readings and lecture topics, and I encouraged students to be critical. After all, nobody saw these but me, and my department head only saw the (very positive) objective scores about how students liked the course overall. Last year, we went online, and suddenly the department head could read page after page of critical comments about course elements. I got an earful. This year, of course, I removed those questions from the survey, instead asking only general questions about impressions of the course. The department head will see the positive scores and be happy with me. But the course will be the poorer for it, and the purpose of the evaluations will have changed.
profjrdn - May 7, 2010 at 10:10 am
Against faculty will, our admin went to all online evals. Rather than “course evaluations” we have “instructor appraisals” and this term has omitted any written comments from students so that there is only the numeric score. Lovely, huh?
jmaddox - May 7, 2010 at 10:43 am
I read this article with interest as I am preparing a report for our faculty on our three-year trial of online course evaluations at our small liberal arts college with traditional classes. I am the Director of IR. We began with and have continued to use a survey identical to our former paper survey, and it is the same for all classes. It also contains an open-ended comments space. Our experience has been just as that described in the paper – that is, essentially we have the same general responses but the response rate that was 80% with paper in class has dropped and stablized at around 60%. However, most faculty report more extensive and valuable comments in the open-ended responses online than on paper, and most appreciate that the results are availabe online promptly. We close all surveys before grades are submitted and hold results until just after all grades have been turned in and posted. The comment about faculty participation being the dominant factor in response rates has also born out for us. Faculty tend to use student comments predominantly, and the statistical responses are used by department chairs primarily to identify serious outliers, particularly among new or adjunct instructors. Some faculty also choose to supplement the electronic with their own in-class questions for students, and they are certainly free to do so. Overall, our experience has been positive, but we continue to look for ways to improve our response rates and refine our information. In the coming year we will work together with faculty and student groups to evaluate our question set and our methods to determine those that suit our needs best, but I doubt we would ever choose to return to all paper.
dboyles - May 7, 2010 at 6:10 pm
“…instructors should learn how to encourage higher student response rates.”And where is the research that indicates that this “encouragement” skews/does not skew the opinions?
eelalien - May 8, 2010 at 1:10 pm
You want “inducement”? The University of Maryland University College (UMUC) who are old pros at 100% online course delivery lock out students just prior to the semester end unless they submit the faculty eval online. Students receive several notices – and later warnings – with clear instructions but if they do not submit by final deadline, they will be prevented from participating in the course, potentially missing assignment deadlines, etc.
reburton - May 10, 2010 at 11:39 am
Im the Market Research Manager at a University in the UK and we face the same problems. I have issues with ‘locking student out’ and Lecturers/Professors standing over students exhorting them to fill in the surveys and ‘don’t forget the results will reflect on the status of the university and thus your degree’ and so on and so for. These approaches are expressly forbidden in the various professional codes of practice for Market/Social Researchers. you american Market Research society would not be happy with this these approaches. And currently HEFCE that runs out National Students Survey has taken to task universities that have offered incentives or have over exhorted their student to fill ini surveys with positive findings – what value is tha data once we know this is happening.In my uuniversity we have the same problem with paper surveys v online surveys. It seems that the academics per se are happier with the paper surveys (perhaps simply because they have a semblance of control and can bias the results – especially if it is they and their performance the students are rating. My feeling is, is that the online survey allows the student to fill it in away from the influence of their professors/lecturers, their confidentiality is maintained, and they have the right to opt out should they wish too.Thats my six penny worth.BTW an someone post the link to this please: the IDEA Center has a list of ideas for doing so on its Web site.(encouraging higher response rates)Thanks (from a Governmentless UK)
reburton - May 10, 2010 at 11:43 am
BTW Sorry for my sloppy typing and the typos
bpconrad - May 16, 2010 at 8:48 pm
Has anyone thought of giving students a $20 rebate on tuition for each online survey completed? (To make this revenue-neutral, institute a “standard course fee” of $20 per course, in addition to any lab or other fees associated with particular courses.)