• October 31, 2014
October 31, 2014, 10:48:10 PM *
Welcome, Guest. Please login or register.

Login with your Chronicle username and password
News: For all you tweeters, follow The Chronicle on Twitter.
 
Pages: [1] 2
  Print  
Author Topic: Validity of RMP feedback  (Read 29202 times)
postmodern
Senior member
****
Posts: 470


« on: November 21, 2011, 1:34:01 PM »

The CHE recently posted an article about ratemyprofessors.com, which indicated that the results can be at least somewhat useful. While I still dislike the website as much as many here seem to, below is a scenario that has caused me to think there is perhaps some validity to it. How to proceed?

Faculty member X has a temporary appointment that could turn permanent. There are indications of problems, but the signifigance of them cannot be determined quite yet. I searched online for some information about a professional endeavor X is involved in and ran across X's RMP ratings page. Yes, I admit, I looked. There were numerous comments over the course of several years. Most had the same (and often quite scathing) negative assessment, and significantly, were in areas I had begun to see problems in too. Certainly this cannot factor into reviews or other decisions, but the parallels are hard to ignore, given the quantity and consistency of reviews. What, if anything, do I do with this information?
Logged
mended_drum
Potnia theron and
Distinguished Senior Member
*****
Posts: 9,477


« Reply #1 on: November 21, 2011, 2:13:32 PM »

What, if anything, do I do with this information?

You find out if the official student evaluations are consistent with the RMP ratings, since those have some standing at your institution.
Logged

"dr. mended_drum don't give a sh!t; she will chew me up like a cobra."
postmodern
Senior member
****
Posts: 470


« Reply #2 on: November 21, 2011, 3:40:47 PM »

Thanks, mended_drum. I've found our students evaluations here are not particularly helpful or valuable. As an institution, our student evaluations tend to be very uncritical. (Averages in most categories are well over "4" out of a 1-5 scale for the college.) While students may talk to me informally about their concerns, they won't usually put them in writing. This has been the case as previous places I've taught at, but not to such an extent.
Logged
burnie
Distinguished Senior Member
*****
Posts: 1,217


« Reply #3 on: November 21, 2011, 3:55:01 PM »

The CHE recently posted an article about ratemyprofessors.com, which indicated that the results can be at least somewhat useful. While I still dislike the website as much as many here seem to, below is a scenario that has caused me to think there is perhaps some validity to it. How to proceed?

Faculty member X has a temporary appointment that could turn permanent. There are indications of problems, but the signifigance of them cannot be determined quite yet. I searched online for some information about a professional endeavor X is involved in and ran across X's RMP ratings page. Yes, I admit, I looked. There were numerous comments over the course of several years. Most had the same (and often quite scathing) negative assessment, and significantly, were in areas I had begun to see problems in too. Certainly this cannot factor into reviews or other decisions, but the parallels are hard to ignore, given the quantity and consistency of reviews. What, if anything, do I do with this information?

In a situation like this I would treat a set of RMP ratings the same way I would a wikipedia article.  Yes, it's a place to start, but you can't cite it and look credible.

Is there a way to find that same info elsewhere - something more credible?

Also, now that you've peeked, you have to be concerned about self-fulfilling prophecy.  You can't administer a peer evaluation or lead a student focused discussion/evaluation (like a mid-term chat) as you might unknowingly and unfairly steer the conversation to get the validation you're looking for. 
Logged

Corporate America wants people who seem like bold risk takers, but never actually do anything.  - Barney Stinson
postmodern
Senior member
****
Posts: 470


« Reply #4 on: November 21, 2011, 4:06:11 PM »

Burnie, I am currently looking for other sources of input. RMP is certainly not credible (even if I suspect it MAY have some vailidity in this case.) I am also concerned about the self-fullfilling prophecy part. I had already brought a couple of others into the loop on evaluating X, even though technically it is solely my responsibility, because I wanted to be sure there was no bias. That is more important now.
Logged
burnie
Distinguished Senior Member
*****
Posts: 1,217


« Reply #5 on: November 21, 2011, 4:31:07 PM »

Burnie, I am currently looking for other sources of input. RMP is certainly not credible (even if I suspect it MAY have some vailidity in this case.) I am also concerned about the self-fullfilling prophecy part. I had already brought a couple of others into the loop on evaluating X, even though technically it is solely my responsibility, because I wanted to be sure there was no bias. That is more important now.

Sorry, I should have asked if you'd considered it not said that you "have" to do anything.  Of course you'd thought of it :)
Logged

Corporate America wants people who seem like bold risk takers, but never actually do anything.  - Barney Stinson
oatmeal
Senior member
****
Posts: 607


« Reply #6 on: November 21, 2011, 6:02:55 PM »

OP--Are you the chair of the department? If you are, you can ask to observe the classes. You can also also ask to view the evaluations. You can ask other colleagues to observe the classes. How is the decision on the full-time appointment going to be made? Are you involved? Is this person a top candidate?

As for "using" RMP information, I do not think you can. It is not a real source and it is not credible, whatever the articles on RMP say. You can only use university sanctioned review processes and protocols. You could use a focus group, perhaps. If you use RMP as part of the process, you will undermine your case and it will lack credibility (even if it is accurate). I would urge you to avoid RMP.
Logged
postmodern
Senior member
****
Posts: 470


« Reply #7 on: November 22, 2011, 4:28:26 PM »

Oatmeal, yes I am the department chair. I have access to evaluations and can conduct observations. Basically, the decision about the position is mine and I am the only one who would directly evaluate X. Therein lies the difficulty because I don't want a decision like this to be made by one individual based on my close interactions and observations. So, I am looking for outside sources of information to provide more feedback. As I previously described, I suspect the student evaluations will not be all that useful.

I would not use RMP as an evaluation tool. Since it seemed to reflect some of the early concerns that exist with X, though, I am trying to find other sources of outside evaluations (besides mine and official student ones).
Logged
sinenomine
Senior member
****
Posts: 612

Located directly over the center of the earth


« Reply #8 on: November 23, 2011, 8:36:51 AM »

Postmodern, can you solicit a separate evaluation from the students, with the justification that the instructor is being considered for a position in your department?  I remember filling out something of that sort when I was a student.

I'll also put in my chime that RMP is not technically reliable, but sometimes spot-on -- we have an adjunct teaching for us right now who had a clear thread of complaint running through his RMP comments from another school, and he's now doing the same problematic thing in my department.
Logged

"How fleeting are all human passions compared with the massive continuity of ducks...."
oatmeal
Senior member
****
Posts: 607


« Reply #9 on: November 23, 2011, 8:50:54 AM »

Oatmeal, yes I am the department chair. I have access to evaluations and can conduct observations. Basically, the decision about the position is mine and I am the only one who would directly evaluate X. Therein lies the difficulty because I don't want a decision like this to be made by one individual based on my close interactions and observations. So, I am looking for outside sources of information to provide more feedback. As I previously described, I suspect the student evaluations will not be all that useful.

I would not use RMP as an evaluation tool. Since it seemed to reflect some of the early concerns that exist with X, though, I am trying to find other sources of outside evaluations (besides mine and official student ones).

As a chair, you should probably check with your dean on what forms of evaluation you can use. Or you can come up with a list, run it by the dean, and then inform the colleague. For example, you should probably get other tenured colleagues to peer-review the teaching. I would suggest asking current students to fill out a special evaluation, with questions that you construct, rather than the regular teaching evaluation. You can survey alumni. I am not sure what other outside evaluations you could use at this stage. If the decision is yours and yours alone, as you state, you can make that decision based on the evidence but also on other factors, such as fit, collegiality and experience. If you have a good working relationship with your dean, you can have a conversation about this. The dean should be helpful...should be. As a chair, I know that these forms of evaluation can take time, but they are sometimes necessary. Good luck.
Logged
oatmeal
Senior member
****
Posts: 607


« Reply #10 on: November 23, 2011, 8:53:50 AM »

You could also ask the teaching center, if you have one, to conduct a review or evaluation of the colleague's teaching.
« Last Edit: November 23, 2011, 8:57:02 AM by oatmeal » Logged
archman
Distinguished Senior Member
*****
Posts: 1,458


« Reply #11 on: November 23, 2011, 12:55:38 PM »

Knowing what I know about RMP, I had a minor flip-out in front of a professor who admitted to me that people on my hiring search committee had looked at my RMP ratings.

Advice to administrators that use RMP, do NOT mention it to other people...

Regarding the article that the CHE put out about RMP last week, you will find that the comments section gives a pretty good impression of what a lot of faculty have observed about the website and their "reviews". The article's authors get pretty blasted, also.
http://chronicle.com/article/Researchers-Rate/129820/
Logged
glowdart
that's a thing that I keep in the back of my head
Distinguished Senior Member
*****
Posts: 5,648


« Reply #12 on: November 23, 2011, 1:04:08 PM »

If you're chair and you're worried about any kind of bias or making this decision alone, then can you ask some of your senior colleagues to observe this person in the classroom?  I would also compile a full dossier of teaching materials -- syllabi, assignments, graded work.  Give yourself as much information as you can. 

Also, are you running a search and is this person a candidate, or are you just contemplating converting the line?  If it's the former, then the situation might resolve itself through the search.  If it's the latter, then I would bring in multiple other voices.

Like sinenomine, I filled out lengthy evaluations as a student when faculty went up for promotion or were candidates for moving from a VAP to a TT line.  Granted, we never did standard course evaluations so these narrative evals took their place, but if there are no rules against this kind of information at your current school, then it might be something to consider. 
Logged
pembleton
Member
***
Posts: 234

...how I long for a grapefruit!


« Reply #13 on: December 10, 2011, 2:12:30 PM »

One thing that's always vexed me about RMP - I've spent thousands of dollars on elective cosmetic surgery in the past year. Do I get a chili pepper? Heck no.

SIGH. I'll be in the freezer with the Haagen-Dazs if anyone needs me.
Logged
brixton
Distinguished Senior Member
*****
Posts: 3,130


« Reply #14 on: December 26, 2011, 6:46:10 PM »

Ummm...Have you considered ways to make your department evals more useful?  Like, say, start with the rmp template?  I'm only partially serious here.  But, if you think your college's instrument is less helpful than rmp, you need to make an appointment with whomever makes the eval and talk about what it doesn't seem to bring out.  This post is frustrating to me, because we have these things called institutional evaluations, and yet chairs and deans don't find them useful.  We spend 15 or more minutes of our final class -- precious real estate time-- administering them, and yet chairs and deans don't find them useful.  Instead they turn to a sight that is pretty easy to double/triple post positive or negative comments.  Save some of that Hagen-daz for me.  I like the stuff with the brittle in it.
Logged
Pages: [1] 2
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.9 | SMF © 2006-2008, Simple Machines LLC Valid XHTML 1.0! Valid CSS!
  • 1255 Twenty-Third St., N.W.
  • Washington, D.C. 20037
subscribe today

Get the insight you need for success in academe.