Revealing the sometimes inelegant process by which research takes place is the subject of the various chapters found in Research Confidential: Solutions to Problems Most Social Scientists Pretend They Never Have (U Michigan Press, 2009 | Amazon, Google Books). The book is edited by Eszter Hargittai, associate professor in the Department of Communication Studies at Northwestern University and fellow of the Berkman Center for Internet & Society at Harvard University. Hargittai leads the “Web Use Project” research group at Northwestern, writes a career advice column for Inside Higher Ed, and contributes to the group blog Crooked Timber.
How would you describe the intended audience of this book?
The book targets mainly junior people — graduate students, general faculty — at a certain level anyone who is interested in understanding at a deeper level than is usually possible for methods write-ups how empirical social science work actually gets done. This means not the pretty write-up that we see at the end of a project, but the ugly reality, because empirical social science research — just like, presumably, research in pretty much any area — has a lot of bumps in it and tricky situations that are sometimes unanticipated, and we are rarely taught these.
What makes this book different from existing methods books?
Obviously methods books are an entire, gigantic literature, and there’s tons of it out there, and they tend to do different things. Sometimes they cover one method in very great detail but still more as “This is how you should do it” as opposed to “These are the problems that are going to happen when you do it.” So again instead of presenting the ideal versions, the chapters here talk about what actually happened when someone was doing the studies. Instead of presenting the ‘hope of the beautiful’ perfect outcome that never happens when you’re actually doing research, this book gives you examples of cases that peers encountered while they were doing work.
One of the things that I focused on when inviting contributors was that I wanted mainly junior scholars. By the time the book came out some of us had tenure, but the point is that the contributors are relatively young scholars. This is important, because another thing that happens with these books is that often they are written by very established people, which is great because these are people who draw on a career’s worth of experiences and that can make them very knowledgeable. On the other hand, if you are a graduate student, is it realistic to assume that you have the same resources and you have the same approach as a super senior scholar? No. So it could be helpful to see how someone very close to your own position handles the situation.
The other side is that often methods books that do go into more depth are usually more focused. Part of the idea here is also to introduce people to numerous methods. What if you don’t even know which method is most realistic for your project? You are getting a behind-the-scenes look at numerous methods at the same time.
I think that’s really valuable, and it coincides with what we are trying to do with ProfHacker, which is not presenting ourselves as experts but just presenting ourselves as people with various levels of experience. People who are going to say, mostly from a first-hand account, “Here is some stuff that I did, here is a challenge that I encountered, and here is how I overcame it.” And it sounds like that was sort of the approach by which this book was generated.
Yes. That’s exactly it. Very much first-hand accounts. The people are experts to the extent that they’ve gone through this experience. Since most are relatively junior scholars, they don’t have an entire career’s worth of experience. But they’re experts to the extent that they’ve had to battle these challenges. They’ve had to come up with solutions. Something came of their work that was valuable, and so why not share that experience? And you are absolutely right that it is similar to what goes on at ProfHacker. It is sort of like lifting the cover off just to say “This is how it actually happened.”
Is there something about scholarship in the social sciences in the year 2010 that makes this kind of book necessary in a way it wouldn’t have been 10, 20, or 30 years ago?
I actually think a lot of this would have been extremely valuable 10, 20, or 30 years ago, too. I was in grad school 10 years ago, and that’s when I thought, “Oh wouldn’t it be wonderful if there was something like this?” But there wasn’t. So at some level, a lot of what’s in here is applicable over time, going way back to methods that have been around for a very long time. On the other hand, there are certain methods that some people cover that are quite new. Some of them just would not have existed because we didn’t have the technology. A few of the chapters address very new tools that we have, either as subjects of inquiry or as tools to gather research. For example, I co-authored a chapter in here with a former student, Chris Karr, on how we used text messaging to collect data from respondents. That just wouldn’t have been a possibility. There were previous forms of it, but we explained why it’s definitely a step up from either beeper studies or paper diaries.
Another chapter, by Dmitri Williams and his student Li Xiong, looks at studying people in gaming worlds online. That’s another topic that maybe ten years ago some of the approaches would have already existed but certainly not much longer before that. People started studying online communities before that, but not on this scale. And so they definitely deal with certain challenges of such a project. And from a completely different angle, we have this one chapter on biomarkers, which seems like a fairly new field for social scientists to get into. It’s pretty funny because that’s a colleague who collects data through collecting people’s spit. [laughter] That actually comes with a lot of complexities, it turns out. Then we have a chapter, specifically on doing surveys online and what challenges that comes with. How can you retain important things that over time researchers have figured out about doing surveys well, but [now we need to apply] them to the online environment?
It seems that in many academic disciplines, with new digital tools and with larger and larger data sets, many traditional methodologies are being challenged and new methodologies are still only emerging without any broad consensus.
In fact, this book barely scratches the surface of some of that actually, but I am working on a second version already that I am going to co-edit with my colleague and friend Christian Sandvig. We are going to address much more of that in that volume and precisely these tools that simply do not exist. How are people going about using them and applying them to questions that may have existed before, but we just didn’t have the tools to go at them in a really sophisticated way?
One of the things that one would do in a book with this kind of approach is to say, “Here are the messy things involved in doing research.” And that’s kind of a dangerous thing for a junior faculty member to be public about. Was there concern about that? Or did people feel pretty confident that it wouldn’t be taken the wrong way?
That’s a very good point, especially because people were so honest. That’s the whole point, right? We don’t usually say that we ran into all these problems. It’s rather embarrassing to admit. I guess maybe people who would have been embarrassed declined my request to take part. Not everyone I asked ended up contributing. I’m not saying that’s why certain people declined, but it’s a possible reason.
Fortunately, I was really excited by how honest authors ended up being. At some level you could argue that maybe it’s because deep down everyone realizes that this is something that everybody faces. It’s not realistic to expect, again, those perfect outcomes. Is there really a senior person who’s truly going to say, “Oh, they ran into problems so they’re not good scholars”? The reality is that everybody runs into these hurdles.
I think people were just being honest about their scholarship, which is great. We cannot take that for granted. It’s too easy not to be honest. But it was really wonderful how honest people were and how forthcoming. I mean, we learn from mistakes, right? Is there something to do, next time, to avoid it? And not everything is possible, but there are little tweaks that are possible.
As I said, every year, I’ve improved my own techniques. Tried to anticipate things, but you can’t anticipate it all. So I would hope that people would recognize that being open about the hurdles just means that we can be more conscious about how to address them and improve on the methods for better outcomes in the future.
In addition to the sequel to this book that you’re working on, do you think there’s a need for other kinds of publications, other kinds of sharing of information in academia with regard to doing the work of being a professor? That’s a really general question, I know.
[laughs] Yeah. I mean I think at that level it’s already like what you guys are doing with ProfHacker. I do think there’s definitely a need. I almost feel like there cannot be enough of this kind of material out there. You guys are addressing it, and there are advice columns elsewhere. I know I read them religiously as a grad student, and I certainly still read a lot of them. I think they can be extremely valuable. If one is lucky, one has great colleagues who also share, but there’s only so much time and there are only so many perspectives you can get from the same people. So I think it is extremely valuable to be sharing and to have diverse voices sharing from the very different experiences.
And what’s also very interesting is that in these cases — in a lot of these cases — experiences cut across specific situations so that experience of someone at a community college may well be applicable to someone at a large research university depending on what the domain is. But those people often don’t necessarily encounter each other that much. So it’s really helpful to have places — and these places will happen online, I think — where people from these very different institutions and also different disciplines can come together and exchange ideas and experiences.