Increasing national college-completion rates, especially among students of color, is a notoriously difficult challenge. Despite decades of research, we have identified very few cost-effective interventions. So imagine my surprise when late last month I read an op-ed in The Wall Street Journal that called on President Obama to promote college attainment among African-Americans by opening “private-school doors for low-income students.”
Incredibly, a study by the authors had supposedly found that the effects of vouchers on college attendance are “unusually large,” especially for African-Americans. For those students, a voucher for private schools “has a much larger impact than does exposure to an effective teacher,” Matthew M. Chingos and Paul E. Peterson concluded in their report, “The Effects of School Vouchers on College Enrollment: Experimental Evidence From New York City,” from the Brookings Institution’s Brown Center on Education Policy and Harvard University’s Program on Education Policy and Governance.
That would be fantastic news for President Obama’s college-completion agenda—if only it were true. Unfortunately it isn’t.
The report makes many critical errors undermining its conclusion. The errors are so numerous and significant, they leave me wondering whether the the other media that are picking it up have vetted its methodology at all before declaring its contents newsworthy.
The study examines the college-enrollment rates of students participating in an experimental New York School Choice Scholarships Foundation Program, which in the spring of 1997 offered three-year scholarships to private schools worth up to $1,400 annually to low-income families. The authors contend that, on average for low-income students, the vouchers produced no overall improvement in college attendance. But they did boost rates of college attendance among African-Americans: College attendance rates were 8.7 percentage points higher when compared to the 36 percent college-attendance rate of the African-American control group not offered the grant. Some of that increase occurred via a statistically significant boost in rates of attendance at selective colleges and universities.
The results were, the authors say, not the same—statistically insignificant—for low-income Hispanic students. The statistical results actually show that the estimated impact of vouchers for African-Americans was indistinguishable from the impact for Hispanics. For reasons I lay out in a paper from the National Education Policy Center at the University of Colorado at Boulder, the estimate for African-Americans may be smaller and less precise than it appears.
Statistical estimates are about trying to find a “signal” that a program works, separating that from the noise that often surrounds it, and in this case the estimate is even noisier—and the signal much softer—than the authors admit. The end result is a false estimate.
There are many reasons that is so. Consider that the overall impact of the vouchers for all students was null, yet the authors claim that their effect for one group was large and positive. Isn’t it surprising, then, that they fail to demonstrate any estimated negative effects for any other group? Moreover, some factors, like college attendance, are not always measured accurately. And, African-Americans and other groups studied exhibit differences (for example, levels of parental education) that could easily account for the findings as well. The details of all of these contentions are contained in my full analysis.
So actually, I agree with Chingos and Peterson: Policy makers and practitioners interested in the effectiveness of school voucher programs should attend to the results of this study. After all, it convincingly demonstrates that in New York City a private voucher program generated no meaningful change in the college enrollment rates of students from low-income families.
