September 14, 2009, 11:00 AM ET

More Money, No More Grads? Hmm ...

Some members of the media are covering the release of a new Canadian study, associated with the Educational Policy Institute, that examines the effects of a financial-aid program on college-going and completion among low-income students. Researchers at the Measuring the Effectiveness of Study Aid Project tried to isolate those effects by examining what happened following a change in student-aid policy in Quebec that increased aid eligibility and decreased reliance on loans. By comparing student outcomes both before and after the policy change, and comparing the outcomes of similar students in Quebec to those in other provinces (where such reforms did not occur), analysts attempted to establish a causal effect of aid.

They conclude that the policy affected access (increasing overall enrollment among students from families making less than $20K per year by 4 to 6 percentage points), and persistence (increasing retention rates by 6 percentage points) but did not affect graduation rates -- at least within the four-year window of time during which graduation was measured.

While noting that the null findings may stem from that short period of observation, the researcher still goes on record with this conclusion: "These results therefore cast doubt on the efficacy of this reform in particular, and of needs-based grants in general, to improve graduation rates." The headline over at Inside Higher Ed reads "More Money Doesn't Equal More Graduates."

This is a distinctly premature and irresponsible conclusion. First, as one of my graduate assistants James Benson pointed out, "if the percentage of college-eligible students that enrolled in college increased by 5 percent, and the persistence and graduation rates remained entirely static, then the program produced a net gain in the proportion of young adults completing semesters and degrees."

Furthermore, there are many reasons why an effect might not be estimated properly in this study. As my colleagues Doug Harris, Phil Trostel, and I explained in a recent paper, a simple correlation between aid receipt and college success is likely to be negative because students from low-income families, in the absence of aid, are for a variety of reasons less likely to succeed. Unless researchers can convincingly account for all of those reasons -- and we argue that very few do -- the estimated effects of aid are likely to look smaller than they really are. This study is not very convincing and really doesn't move far beyond a correlation, for many reasons. For example, as another graduate assistant, Robert Kelchen, indicates:

1. The comparison groups (Quebec vs. other provinces) have very different rates of financial aid take-up prior to the reform. This calls the validity of the comparison into question. It's also too bad the researcher didn't see fit to post his tables on the Web site, since we cannot see whether the differences post-treatment are significant.

2. Quebec saw increases in the enrollment rates of high-income students following the reform, in addition to increases in the enrollment rates of low-income students. If financial aid was the real driver, it shouldn't have affected the (ineligible) high-income students.

These are but a few examples -- if a full research paper (such as would be submitted for academic review) was available, I bet we'd have more concerns.

This is a case of the press jumping the gun and running with a story, and a headline not supported by the empirical work done by the researchers. We're in a recession, and aid programs cost a lot of money. We do need to know if they work, and in particular if they are cost-effective. But the estimation of impacts should be done more carefully, and results discussed in a much more responsible manner. Sexy but uninformed headlines will do little good -- perhaps even casting a shadow on an effective program, reducing its ability to maintain funding. All of us studying financial aid have an obligation to do much, much better.

E-mail

Comments

1. 11167997 - September 15, 2009 at 06:26 am

An exemplary case of involving your students in a public assessment of propagandistic research. Right-on, Sara! On, Wisconsin!

Cliff Adelman, Institute for Higher Education Policy

2. primaryovertone - September 15, 2009 at 10:00 am

I also find it interesting that the study concluded after four years when it is so very common for students (especially those with financial hardships) to finish their degree program in five or six years instead of four. That is why graduation rates are caluated on the 150% of program system for the federal government.

3. goldrick - September 15, 2009 at 10:11 am

To be fair, when you've got just 4 years of data collected so far, it's ok to write about it. Imagine, from the researcher's perspective, being forced to wait 6-8. Doesn't make much sense. The issue isn't about the time he waited or didn't wait, but about how he interpreted his results-- and how those results were described in the press.

4. primaryovertone - September 15, 2009 at 12:27 pm

goldrick,
I disagree. This group has to know that the media is going to pick up what they have written and is going to assume that they have completed enough research to draw valid conclusions. An Educational Policy Institute is writing about education, any member of the press is going to assume they know what they are talking about. Publishing CONCLUSIONS two-thirds of the way through a study is silly. Unfortunately this silliness is exactly why educational policy makers in Washington are so missinformed. They read articles like the one mentioned and make assumptions based upon the assumptions of a member of the press who assumed that the assumptions of educational experts were based upon facts and not assumptions.

The truth is that if the four-year graduation rates are the same as previous graduation rates that the six-year graduation rates will be higher. The only question is "How much higher?"

5. goldrick - September 15, 2009 at 01:29 pm

Since when does the media "assume" anything? Not since the 1970s. The Media has a responsibility to ask questions, such as "has this work been vetted? If so, by whom?" They should note when a place is a contract shop, rather than academic organization. The reporter should also be trained sufficiently in statistics to be able to know if numbers make sense. The writer/researcher also has the responsibility to stay true to their data and not over-reach in conclusions. But some of these folks work for places where they are paid to make bold statements, and it's the media's job not to simply copy/paste their press release into an article, and instead to do their homework. Otherwise, why don't we just have a "newspaper" where researchers just upload their press releases?

6. primaryovertone - September 15, 2009 at 02:41 pm

goldrick,
Have you actually read anything on cnn.com or foxnews.com? While you may feel that media organizations have certain responsibilities, they have different ideas. Why else would we see "scientific" reports on a daily basis concerning the consumption of alcoholic beverages which are diametrically opposite to each other. There is no question of vetted work vs. non-vetted work in the news media. That is the concern of professional journals. The news media is working in a twitter world where everything must be up to the minute. They don't have/take the time to check facts. If they did take time to check facts one of the other news organizations might scoop them.

7. goldrick - September 15, 2009 at 09:45 pm

Oh come on-- that's setting an absurdly low bar, one based on the worst organizations. There are plenty of reporters still out there doing their jobs, online and off. Deb Viadero at Ed Week is a good example.

8. primaryovertone - September 16, 2009 at 08:47 am

goldrick,
Your good example is a reporter for Education Week. This is a journal about a profession for a profession and thus a professional journal.

The sad fact is that this article appeared in Inside HigherEd which purports itself to be tied to higher education. While I do feel that the journalist oversteped his bounds I feel that more responsibility falls on the shoulders of the group who jumped the gun by drawing conclusions on a research project two years before the research was done and publishing their conclusions. This is like handing a diploma to an undergrad in their junior year of college and telling them that you still want them to finish their senior year.