I’ve been away for the past week so haven’t had a chance to welcome the newest contributor to Brainstorm, Sara Goldrick-Rab. Welcome! Sara does terrific work and was nice enough to invite me to Madison recently to speak to her students and colleagues. The blog is lucky to have her.

She’s also open to conversation so I know she won’t mind if I disagree with the thrust of her recent post about a policy brief I co-authored, Diplomas and Dropouts. The paper’s arguments go something like this:

(1) Many colleges have terribly low graduation rates.

(2) When you compare colleges with similar admissions selectivity and incoming student academic profiles, as defined by Barron’s, you find huge variance.

(3) Therefore, it’s reasonable to conclude that while graduation rates are clearly influenced by external factors like students’ academic preparation and aptitude, they’re also influenced by what colleges themselves do.

(4) Therefore, it’s reasonable to conclude that colleges with the lowest graduation rates, compared to their peers, could do better.

Sara objects that these conclusions require a proper accounting of the tremendous selection bias inherent to a diverse, market-based higher education sector, and that our method for doing this falls well short of “the gold-standard tools needed to figure out what works (say, a good quasi-experimental design). Important student-level characteristics (socioeconomic background, high-school preparation, need for remediation, etc.) aren’t taken into account. Nor are many key school-level characteristics (e.g. resource levels and allocations).” That’s true. But frankly I don’t think it undermines the paper’s conclusions, nor do I think we should wait for such research to be performed (and hey, why hasn’t it?) before crafting policies designed to improve graduation rates. Here are some of the reasons why:

I’m perfectly comfortable assuming that colleges make a significant difference in the educational lives of their students. Does anyone really want to take the other side of this argument? When the subject turns to outcome measures like graduation rates, colleges often come perilously close to arguing from their own inefficacy: “Don’t blame us — we don’t make any difference!” Yet whenever college budgets are cut, everyone says that students will suffer. Well, you can’t have it both ways — if you matter, then you matter. People like Sandy Astin have noodled around with regression analysis of six-year graduation rates and have run the R-squared up to around 60 percent based on external factors like student test scores, funding, etc. It’s easy enough to do with IPEDS data and SAT/ACT scores by themselves get you a lot of the way there. But 40 percent of variance is still a lot of variance. If all we had to work with was the 40 percent, we could still help tens of thousands of additional students earn degrees. Sara says we should control for things like school-level resources in these discussions. I disagree. Let’s say we have two colleges, identical in all ways including their student body. The only difference is that College A gets twice as much funding as College B, and as a result College A graduates many more students. That’s not a mitigating factor to be explained away — it’s an opportunity to help students, a problem to be solved.

Because the analysis says little about why some colleges perform better than others, Sara says, “I don’t see how this approach is moving the ball forward — sure it gets peoples’ attention, but it’s not compelling to the educated reader — the one who votes and takes action to change the system. Moreover, it doesn’t get us any closer to the right answers, or provide any confidence that if we follow the recommendations we can expect real change.”

I’m going to go out on a limb and say that the average educated voter has no idea that many colleges routinely post graduation rates of 40 percent, 30 percent, and less. The world is run by college graduates who mistakenly assume that their postsecondary experience was typical. So if all the report does is get people’s attention, I’d be thrilled. The first step to creating political momentum around a problem is convincing people that the problem exists.

Moreover, I don’t think good public policies need to presuppose the “right answers.” In fact, that kind of presumption often leads to bad policy. Colleges and universities are extremely complex. Institutions with similar missions and bad outcomes may have those outcomes for entirely different reasons. Maybe the issue is leadership, or culture, or faculty, or administration, or resource allocation, or pedagogy, or counseling, or program design. I don’t think the government should be in the business of telling colleges how to succeed with their students. But I do think it should be in the business of holding colleges accountable for success.