next »

[scampster]

So last year I participated in a survey by an institution with an active ADVANCE grant from NSF regarding women in STEM fields. I got an invitation to participate in a follow up survey this year. I'm always happy to add my datum to the pile.

Besides being a poorly designed survey (half the questions didn't make sense - what are they teaching the psychology students writing this stuff?), they wanted information on where I had accepted a position and then they asked all these questions relating to the accepted position. These questions were all loaded questions like:

"Do men have an easier time in your department than women?"
"Is your department hostile towards parents?"

While I know the survey is confidential, there is no way in hell I am giving the name of my employer and letting it be attached to frank survey responses. Maybe I'm wrong, but I imagine most women would feel the same way - it's one thing to collect anonymous information about the percentage of women who feel these things to be true in their departments, but another to attach those feelings to specific departments (not that I believe that is the intent, but it feels sketchy to me to have information about both in the same survey).

FWIW, I don't have any bones to pick with my departments on these issues, but I got halfway through the survey and got fed up with the badly posed questions and exited out.

Survey FAIL.

[kedves]

There are ethical and methodological problems with the questions you describe.  It would have been better if they asked a more general question about which type (from a set of choices) of institution yours is.  Confidentiality and anonymity are very different things.

Did you give any feedback directly to the survey administrators?  There is usually a comments box at the end, but you could also send an email.  It's useful information--and better for them to consider now than when it's published and the criticisms are unexpected.

[dellaroux]

What IRB passed the survey, anyway?

Or did they?

[scampster]

Della - I was kinda wondering that! I think it did go through an IRB (I seem to recall some front matter to that effect).

Kedves - I was thinking of dropping the survey adminstrator a note, at least to explain why I personally wouldn't be filling it out. If they think I am being silly and they have gotten plenty of other responses to make the absence of mine unimportant, then good for them!

[vardahilwen]

This is precisely why I spend a lot of time teaching my students about leading questions, loaded questions, and why they're usually bad.

[prof_smartypants]

Ugh. I took the best survey class ever and did a survey for my dissertation. That survey, scamster, makes me very sad.

I'm doing a survey module in my undergraduate research methods. I'm going to use those two questions as examples of bad survey questions.

[karmann]

I think these people spoke at an assessment conference I just attended!  They were conducting research on women in STEM fields, repeatedly mentioned an NSF grant, and nearly everyone in attendance had this "WTF?" look on their face the whole presentation.  First, we had no idea what this had to do with assesment.  Then, yeah, the whole thing seemed horribly designed, even to someone like me who has never written a survey and never will.  I wonder if these were, in fact, the same people. 

[locutus]

I was marginally involved with a similar survey a few years back. We had similar problems in the initial design, wording of questions, concerns about anonymity. It sounds to me like they're trying to have their cake and eat it too, with regard to detailed position information and blunt feedback on working conditions. They didn't make these separate forms?