by

How to Lie With Education Data, Part 2

On Tuesday I wrote about a tongue-in-cheek post at Forbes that tried to make a point about the cost of college. I argued that the piece failed readers by falsely equating cost and value, among other problems.

A subtler but more worrisome example of misleading data appeared on Monday at the top of an article in The New York Times about the predatory activities of some debt-settlement companies, two of which stand accused of charging exorbitant fees to student-loan borrowers for free federal debt-relief programs.

The article’s authors set the stage for their story with the following “statistics” about student debt:

Student loan debt hovers at more than $1 trillion, a threefold surge from a decade ago, and a record number of college students who graduated as the financial system nearly imploded have an average debt load of more than $20,000.

More than half of recent graduates are unemployed or have low-paying jobs that do not require that expensive college degree. Some Americans, including baby boomers whose savings were devastated by the financial crisis, are still struggling to pay off their student loans well into their 50s.

There are four discrete data points in here, each of which is misleading in a different way. Let’s break them down:

1. “Student loan debt hovers at more than $1 trillion, a threefold surge from a decade ago.”

That oft-cited statistic comes from the most recent “Quarterly Report on Household Debt and Credit,” by the Federal Reserve Bank of New York. The exact figure for the first quarter of 2014 is $1.11-trillion, up from $260-billion in the first quarter of 2004 (which is actually a fourfold increase).

While the quoted statistic is true, it lacks context. How does the increase in student-loan debt compare with other types of debt? How many people are counted in the figure, versus the 2004 statistics? How much of the increase can be attributed to inflation? What reasons are there for the increase?

Blanket statements that purport to explain a phenomenon with data in fact do just the opposite: They further obscure and cloud readers’ understanding of an issue.

2. “… a record number of college students who graduated as the financial system nearly imploded have an average debt load of more than $20,000.”

That clause is a little unclear, but it seems to be saying that a record number of college students graduated as the financial system nearly imploded and that they had an average debt load of more than $20,000. It’s true that a record number of students graduated in the midst of the financial crisis; in fact, a record number of students have graduated every year since 1994-95. It’s also true that the average debt load—for students who graduated in 2009 and who borrowed for college—was more than $20,000.

What’s misleading about those statistics is the implied connection between graduating in the midst of the recession and average debt from student loans. As the Project on Student Debt reported in its October 2010 report about the Class of 2009, “The six percent increase in average debt at the national level is similar to the average annual increase over the past four years, despite the recent economic downturn. It is likely that the Class of 2009 took out the bulk of their student loans before the recession began.”

3. “Some Americans, including baby boomers whose savings were devastated by the financial crisis, are still struggling to pay off their student loans well into their 50s.”

First of all, it is hard to determine if someone in their 50s is “struggling to pay off” loans because that requires knowing how much of a person’s monthly income is being devoted to student-loan payments by age, a cross-section of data that we don’t have.

What do we know about how many people in their 50s are still paying off student debt? According to a survey published by the Federal Reserve Board in July 2013, 7 percent of adults ages 50 to 59 were carrying their own student-loan debt, while 12 percent were carrying student-loan debt for their children, and fewer than 1 percent were carrying student-loan debt for both themselves and their children.

So are some Americans still paying off student loans well into their 50s? Sure, but it’s a small group, it’s unclear whether they’re struggling to do so, and we lack context as to whether this is a change from historical norms. By comparison, for Americans in their 40s, according to the same Fed data, 14 percent were carrying student debt for themselves, 5 percent had student debt for their children, and 1.5 percent had student debt for both.

4. “More than half of recent graduates are unemployed or have low-paying jobs that do not require that expensive college degree.”

This may be the most problematic statistic. The data here appear to come from another report by the New York Fed, which used Census and Labor Department data to look at the employment of recent graduates, defined as people ages 22 to 27 with a bachelor’s degree or higher, from 1990 to 2012.

The report says that the unemployment rate for recent college graduates in 2012 was 6 percent, down from a peak of 7 percent in 2010. For some context, that’s higher than the unemployment rate for all college graduates (4 percent) at that time, but lower than the unemployment rate for all workers (just under 8 percent)—and significantly lower than the rate for workers ages 22 to 27 without a college degree (more than 13 percent).

If 6 percent of recent college graduates are unemployed, then for the Times’s statistic to be true, more than 44 percent of recent college graduates would have to be employed in low-paying jobs that don’t require a college degree. And sure enough, at first glance the Fed report says 44 percent of recent graduates are underemployed, defining underemployment as those who are in a job that, according to a majority of people who hold it, doesn’t require a bachelor’s degree.

However, the devil is in the details. The report explains that the “underemployment rate” is calculated based on the number of graduates underemployed divided by the total number of graduates employed. In other words, it’s not that 44 percent of all recent graduates are underemployed; it’s that 44 percent of employed recent graduates are underemployed. That makes a big difference, as we’ll show.

According to data released recently by the Education Department’s Baccalaureate and Beyond (B&B) study, which tracks the outcomes of students who graduated with a bachelor’s degree in 2007-8, 69 percent were working approximately four years after graduation. An additional 11 percent were both working and enrolled in postgraduate education (the Fed report specifically states that it excludes anyone “currently enrolled in school”), 6 percent were enrolled and not working, 7 percent were unemployed, and 8 percent were out of the work force (the figures don’t add up to 100 percent due to rounding).

Let’s assume for a moment that the B&B study, which provides a snapshot of just one class, applies to all graduates ages 22 to 27. If 69 percent of recent graduates were employed and not in graduate or professional school, and 44 percent of those employed graduates were underemployed, then only 30 percent of all recent college graduates were underemployed.

But there’s another problem here. The Times article wasn’t just talking about underemployed college graduates; it was talking about those who were in low-wage jobs that don’t require a college degree. Helpfully, the Fed report divides underemployed recent college graduates into two categories: those with “good non-college jobs” or with jobs that don’t require a college degree but pay an average of $45,000 or more per year; and those with “low-wage non-college jobs” or jobs that pay an average of $25,000 per year or less. (Obviously there’s a group in the middle.)

According to the Fed, in 2012, 36 percent of underemployed recent college graduates were in good jobs, while about 20 percent were in low-wage jobs.

Let’s apply those numbers to the underemployed figure. If 20 percent of the underemployed recent college graduates were in low-wage jobs, and 30 percent of all recent college graduates were underemployed, then only 6 percent of all recent college graduates (20 percent of 30) were in low-wage non-college jobs. Add that to the 6 percent of recent college graduates who were unemployed, and it turns out that only 12 percent of recent college graduates were either unemployed or employed in “low-paying jobs that don’t require that expensive college degree” (never mind defining an expensive college degree).

Again, those calculations are based on data from a single class of graduates who entered the work force just before the recession, but even if the numbers are somewhat higher for those who graduated right into the recession, it would still be well below “more than half” for all recent college graduates.

This is obviously a serious mathematical error, and we all make mistakes. In fact, the Times corrected a previous version of the article that had an even more glaring error, stating that “More than half of all college graduates are unemployed.” But even the other statistics in the article, which are mathematically accurate, suffer from a lack of context and, more important, from a lack of attribution. (Rachel Abrams, one of the reporters who wrote the article, declined to comment for this post.)

The problem, though, isn’t really just with this specific Times article. Journalists are often guilty of reporting data without subjecting that data to the same scrutiny we would apply to other sources of information. We use data as context, instead of providing the necessary context and attribution for the data we cite. And without attribution, readers are left with no way to verify or find context for the data on their own. The average reader would be forgiven for assuming that data written in this way is Truth.

The truth (with a lower-case “t”) is that all data are open to interpretation. Just like a historical “fact,” data appear different when looked at from different perspectives, within different contexts, or from different sources.

Unattributed and unverified data are how mistakes get entered into the public record. Next thing you know, someone will come along and cite The New York Times as the source of a statistic that says that more than half of college graduates are unemployed or in low-wage jobs. And that’s just not true.

Return to Top