The Booth School of Business at the University of Chicago has had written course evaluations for decades, carefully crafted by their world top 5 statistics faculty. The data are published for every professor and every course but with no required responses of any sort---let the buyer beware for the student side, and hire bright people and leave them to heck alone on the faculty side. The results have been sustained maintenance of huge MBA numbers, growing doctoral numbers, and steady ranking in the top five across those decades. ON THE CONTRARY---the above "during the course" criterias are pitiful, intrusive, out of date, and silly. The "after the course" criteria confound with dozens if not hundreds of stronger variables--they are statistically irresponsible as provided above. The "the intructor" variables strangely test for everything except relevant-to-instruction attributes of the instructor, his/her goals/interests, the degree-of-forced-ness of the course upon him/her by possibly sick (with power) administrators, and like real world determinants of course/instructor/teaching effectiveness. God help the institution who applies any of the above measures--I suggest choosing better institutions of higher learning.

GOD'S SENSE OF HUMOR--the NYTimes last week has a research journal article summarized showing that "good writing" harms retention of points. I suspect that a well crafted study would show that "good teaching" as in any of the points above, harms retention of points, too. Go figure!!! Life is complex.

These are stupid measures. Instead of flakey or statistically suspect crap, we should track the number of times a professor is late to class, the number of times a professor cancels class, the turnaround time for returning a graded assignment, the percentage of a required textbook that is assigned to students, the availability of instructor contact information, the percentage of documented course objectives that are taught by the professor, the clarity of the professor's language of instruction and so on. Evaluations are a blunt tool and should merely be designed to catch the grossly incompetent so that he or she can be fired or not reappointed.

A 102 level course implies it's on introductory material - so why is it considered a good thing that the instructor have a deeper immersion in the subject area (i.e. doctorate)? Shouldn't we care more about the instructor's ability to get the introductory material (generally broad but shallow) across to the students than whether or not the instructor can delve deeply into some esoteric aspect of it?

So much to measure; so little time.

I had to check the calendar to make sure that it's not April 1st. In what world would these be useful measures of course quality?

Look at Why it Might Matter on class size. It says "students seem to learn better.." That implies two things: 1) there is a way to measure student learning and 2) if students "learn better", the course is higher quality.

Therefore, instead of coming up with a long list of things that "might matter", why don't we just measure student learning?

Now if this list was presented as useful measures for marketing a course...

Thanks for this helpful survey of options -- the good, the bad, and the ugly. I agree that few of these "measures" actually evaluate what students actually learn.

However, while I appreciate the candid recognition of the limits of each approach, the net effect is "none of these work," which is not true. Yes, "statistically reliable conclusions require a few years of data," but we can develop this over time. Yes, standardized tests "might not capture students' ability to synthesize and apply knowledge" -- but then again a good test might and this option should not be eliminated without further consideration.

I am a bit intrigued that virtually none of the proposed measures evaluate the contribution of the introductory course to the education of the non-major. True, one cannot evaluate the contribution of one specific course to students' performance four years later. But we presumably can, at the conclusion of the course, evaluate some contributions of the course to general education outcomes. If we can't describe and measure what we expect students to gain from the course, what is the point of including the course in the general education requirements? Yes, I know that not all learning can be measured and some courses have wonderful, serendipitous effects on some students, but we don't build a curriculum in the simple hope that some students will somehow be effected in some unspecified but wonderful (and no negative) ways that we cannot anticipate. There is some reason this course is included in the curriculum for non-majors. What is it? If the answer is "because everybody ought to have a basic psychology course," the next question is "why? What do we think the benefits of taking such a course should be?" Once we have an answer, we have learning outcomes and can evaluate how well the course achieves them.

I see nothing to indicate the instructor has any practical experience in the field? Is not the study of psychology as well as many other fields (e.g. business, social work, nursing, education) enhanced when the instructor is not only academically qualified but also an experienced professional in the field?

The information you presented would be dramatically improved if you also had a measure of validity attached to both the cklaimed benefit and the claimed criticism.

Research based assessment conclusions are helpful. The ease of obtaining data ought to be relevant only in that context. The fact that data is more difficult to atttain is mitigated if it is useful. Easy to obtaion data is usless if it's just populatiry or lore.

There is a HUGE BLIND SPOT in these criteria:

Can the instructor TEACH?
Are media used EFFECTIVELY?

I don't care how many degrees you have, where you went to school or whether someone has "computer-assisted learning". If the instructor doesn't focus on teaching and use the media effectively (I've seen people write on a whiteboard so small that you can't read it 10 feet away), they SUCK as an instructor.

Quality is NOT all academics. It's possible for a highly educated instructor to stink at teaching in any medium. Until faculty accept that they need to devote energy and attention to their teaching and learn how to use classroom media effectively, we will still have low quality courses taught by PhDs.

This is just plain dumb. Can we move beyond all of this assessment and discuss imagination and creativity in teaching and learning?

The list includes the statement, "All else being equal, students seem to learn less in lecture-heavy classes." Any data to support that? No.
It's true that a lecture-heavy class taught by some monotonic droning bore is deadly, but some classes are necessarily more lecture heavy than others. How much student input are you going to get, or should get, when you're covering the Great Vowel Shift in English?
The valid points of this list are pretty self-evident, and the rest sounds like pablum coming out of a school of education.

Look at comment #3 for the administrative view of faculty evaluation. An utter ignorance of the mostly-intangible characteristics of a good teacher underpins many faculty rating instruments. This compulsion to measure things is another product of the galactically misguided desire to allow market forces to govern academics. Students and colleagues can tell a person who the good teachers are, and that rarely rests on amnirov's criteria (see comment #3). Late arrivals can be annoying, but forgivable if the person is a great scholar/teacher. The bottom line, which will eternally escape the measurers in academia, is that most of the best characteristics in teaching cannot be measured except by the before and after of students who take a particular course taught by a particular teacher, and even then it is difficult to articulate what those characteristics were.

Bully for #12. Craigchicago is dead on. A lecture by a great scholar/teacher is always good, and it assumes that the students accept some responsibility for learning from that teacher. I've had intensely boring classes, but because I was paying tuition and didn't want to waste my money, I learned something from the droning professors as well as those who were great teachers. WHERE I LEARNED NOTHING was in a class with a sparkling performer whose knowledge of the subject was google-shallow. This latter model seems to be what is preferred by most administrators, begging the question about what their real motivation is, educational or financial.

By the way, after ten years of college (and now 16 years of teaching), I must assert that anyone who doesn't look at a class cancellation as a gift from heaven is psychotic.

Both #12 and #13 seem to be in love with the "great teacher" image. #12 cannot even think about if or how to get student input when covering the Great Vowel Shift in English. That says it all.

I don't much care if someone is a great teacher vs an average or mediocre one. There isn't a mechanism in my contract to reward excellence in the classroom, but there is a need to punish incompetence. Late arrivals are not just annoying, after a certain number of late arrivals, they are in fact in violation of the contract. Similarly, students have the right to know that if a professor is assigning a $200 text book that the book will be used. Chairs need to know that faculty are marking and returning high stakes assignments etc. If a reader has a problem with that, all I can say is thank christ he or she isn't teaching alongside me because I don't want to deal with his or her prima donna snowflake BS, or since I'm tenured, I'd be sure to tell him or her to his or her self-important face just what I thought about "Great scholar/teachers" being late to every single flipping class. It would not be pleasant for anyone other than me. I'd have a riot telling that blockhead off.

"In many courses well-designed software seems to improve student learning."

I'm glad to the word "seems".

I'm surprised noone has mentioned how quickly a course fills up. When I was an undergrad we knew who the good teachers were and their classes filled up very quickly. The instructor to be named classes were the ones that you got stuck in when all the good classes were filled.

Another point that is not raised is the difference in teaching undergraduates, and training of graduate and professional students. Someone who is good at training graduate students, may not be good at teaching undergrads. Should we use the same measuring stick for all types of instruction? There are even a few researchers that are terrible teachers across the board, but do great research, bring in big money, and have great students who go on to be successful.

The after the course section seems like it would apply more to a department, than to a single course. I would argue that differences in after course outcome would better correlate with the self selection of motivated students into the better classes. A good tenured teacher's class would likely have a choice class time, students would be aware of who the good teachers are, and motivated students would likely be overrepresented in this class when compared to the students in a class taught by a last minute adjunct, who has a less desirable class time, and a class filled with students who couldn't get into the good class. Even if the adjunct is a fantastic teacher, it is unlikely that the students in his class will outperform those students in the tenured prof's class. There are also the out of class benefits of taking the tenured prof's class, such as recomendations, understanding expectations in later courses, school specific advice, an office for stopping by in or out of office hours.

As usual, the comments are telling for the oversimplifications (e.g. all administrators love shallow but sparkling performers as teachers; This is usually written by people who can't teach their way out of a paper bag) the disdain for any way to measure learning/performance (difficult but not impossible), the justifications for practices that would be impossible anywhere else (try telling your boss at a regular job that you are late every day, but it's OK because you are brilliant...), etc. To #15: there isn't a mechanism in your contract to reward excellence in the classroom? I bet you are teaching at a Research 1 institution.... Lots of other schools have those mechanisms and the reward is both monetary (above average raises) and more intangible (students love you).

I liked amnirov's comments. I completely agree. I audited a course that rarely started on time, had classes canceled or 'rescheduled' several times at the last minute, and we even had a sylabus change midway through the semester. Being treated like a peasant really made me resent the class. It made learning in that environment much harder, I can't imagine the annoyance those students who were taking it for credit experienced.

#14 a_voice makes a good point. Rather than using the theory of phonological space to show students how the GVS probably took place (linguists aren't totally sure how but they've made a good guess), I could ask students how they feel about the GVS and then query them about why the GVS does or doesn't deal with their du jour identity issues. Thanks, a_voice. I'll do that next time I'm covering diachronic sound change in English.

Oh I'm not even a teacher but that last comment by craigchicago was hilarious. I personally find the GVS off-putting.

I'm not sure where all the passionate hatred in the comments comes from.

My perspective is that faculty ought to provide the opportunity to learn. If the required material is covered in a competent fashion (i.e., you can learn if you pay attention and do the required outside work, and you have reasonable access to ask questions) the evaluation is done.

One difficulty with the measures presented above is that of separating entertainment from teaching. Students might like coming to class, but I've also seen cases where instructors substitute entertainment for competence. I've watched claimed star instructors in action, and in many cases, I found their explanations of difficult concepts to be mediocre.

#1, #2 & #3 are a little too arbitrary for my liking. How can you use essentially demographic criteria to determine a course's quality? It seems to me that both measurement columns are equally plausible. What if you turned the tables? For example...

Is the professor old or young?

Why it might matter - Younger professors are more energetic, enthusiastic, closer to most of the students in age and have a more recent and better memory of the challenges they face.

Limits of the measure? Older teachers can establish a deferent rapport with the students more easily. They may also be more experienced.

jenwrye, your points are well taken and show the general silliness of so many of the list's items. Might I suggest another question:

Is the professor hot?

Why it might matter -- students will pay better attention to the professor.

Limits of the measure -- students will pay better attention to the professor.

This article is really stupid....

Human sexuality is what psychology professors specialized in. a high percent of psychology majors were themselves abused...