Misgauging College Performance With Graduation Rates?

Today, we'll be discussing a new report from a senior researcher at the U.S. Education Department, that contends that student-graduation rates are not a reliable measure of a college's performance.

This finding is politically controversial because it comes at a time that policy makers are weighing proposals that would penalize or reward colleges based on their graduation rates.

Is the report correct? If not, why not? If so, are there better gauges of performance that could be used?

Our guest today is the author of the report: Clifford Adelman, a senior research analyst at the Education Department. Mr. Adelman has worked at the department for 24 years, under both Republican and Democratic administrations.

Thanks for joining us, Mr. Adelman.

First, I'd like to thank the U.S. Education Department's Institute for Education Sciences for supporting the research, and especially Susan Sclafani, counselor to Education Secretary Rod Paige, for bringing the results of the research to our constituencies in higher education and the general public.

To start off, I'd like to explain what this piece of research is about and what it is not. It's called "Principal Indicators of Student Academic Histories in Postsecondary Education, 1972-2000." It is a reference work based on three grade cohort longitudinal studies conducted by the U.S. Department of Education's National Center for Education Statistics. Think of them as the histories of the high school classes of 1972, 1982, and 1992 followed out until the students were in their middle-to-late twenties. The core evidence consists of the college transcripts of the large national samples of students in these studies.

"Principle Indicators" covers topics of geography, demography, access and participation, attendance patterns, attainment, majors, grades, and remediation, and with trends across nearly 30 years of student histories. I hope we can talk about a variety of these topics this afternoon.

The report is a large, sweeping background tapestry, not a topic-focused or specific population-focused study.

In reporting on my study, The Chronicle focused entirely on the issue of graduation rates from four-year colleges: the difference between student-based graduation rates and institution-based rates.

I want to be clear that graduation rates for degree-seeking students are the bottom line of success and what institutions contribute to student momentum toward completion is a legitimate part of the accountability equation.

But with nearly 60 percent of traditional-age undergraduates attending more than one institution--and with 20 percent of bachelor's degree recipients who started in a four-year college earning their degrees from a different four-year college--it's clear that students are not taking a straight path to graduation. Those who would design accountability systems should take these and other indicators of student mobility into account.

My role and the role of the report is to document the kinds of indicators and formulations that might be investigated. If you want to read the report, it's available online at http://www.ed.gov/rschstat/research/pubs/prinindicat/index.html.

For example, in a recent study conducted by the National Center on Education Statistics, one out of nine students interviewed did not tell us about at least one school they had attended as undergraduates. How do we know this? Because the college transcripts for these students included transfer credits from schools never mentioned in the interview. Such information is obviously important in understanding the complexity of attendance patterns.

On the matter of earning degrees, transcripts are only a little more accurate than interview-based data. Most people don't lie about degrees.

However, on matters of course work, remediation, college credits, grades, and dates of enrollment, you can't beat the transcripts. We had one survey in which 15 percent of students interviewed reported taking remedial courses, but the transcripts showed that something like 45 percent actually did take these courses.

As is the case with all three of the center's longitudinal studies completed to date, we took a sample of high schools and then randomly selected students in those schools. In this case, the students were in the eighth grade in 1988. In the last year of the twelve-year study, we asked the students where they had received their postsecondary education and then we got the students' transcripts from those schools. The institutional response rate was 90 percent and student anonymity was guaranteed.

I want to make clear that I am not the one who "arrived at the sample." It was a process set in motion by the center and carried out by the contractor for that task--the Research Triangle Institute.

Do colleges' graduation rates have any value at all?

Clifford Adelman:
    Yes, the rates are a good measure for institutions to use to evaluate themselves.

Elite colleges don't worry about their graduation rates. Students don't walk away from a Harvard University degree. Selective schools, which include most of the flagship state universities, might be concerned on an intra-state basis because they ultimately answer to state legislatures, which are justly concerned with cost efficiencies.

Non-selective four-year schools are non-selective for a reason: they account for 80 percent of undergraduate enrollment, and cannot expect high graduation rates. Many of them are commuter schools, and loyalties to any commuter institution in our society are tenuous. (I both taught and served as an associate dean in such institutions, and know from those experiences that students were stopping out and changing schools at very visible rates.) It's not the institutions' fault. The point of "Principal Indicators" is that system-wide graduation rates are more important.

Despite their value, graduation rates are not a measure of learning. One would want to know just as much about how current and deep is the knowledge that a degree recipient--whether from a community college or a four-year college--brings to the labor market and to the community.

There are proxy meausres of successful course completion that one might use. "Principal Indicators" has some of that information, but its companion document (which hasn't been issued yet), "The Empirical Curriculum: Changes in Postsecondary Course-Taking, 1972-2000," offers a great deal more.

A good example from a community college: if the dean of instruction would say something like "81 percent of our A.A.S. recipients in Allied Health Sciences have completed two or more courses in radiologic technology, hematology tech, and medical imaging, and have passed a comprehensive computer-based examination on medical information systems," that dean has said a great deal more about the knowledge the community college has imparted to the students who bring it into health-care settings than merely counting diplomas.

Using students who were in American high schools in the 12th grade, our longitudinal studies offer a number of more accurate denominators, and "Principal Indicators" illustrates what happens when you change the denominator.

Given what you, Mr. Adelman, know from your extensive research, what kind of metric would you suggest? How could we give helpful information on graduation?

Clifford Adelman:
    There is no question that an accurate metric of graduation is helpful to everybody.

The first item to consider in such a metric is what we mean by "graduation." If all we mean is a bachelor's degree, then we miss people who seek--and attain--associate's degrees. If all we mean is an earned degree within X years of entering postsecondary education, then we miss people who are still enrolled and pursuing degrees at the end of X years.

My colleagues at the National Center for Education Statistics use a combination of degree attainment plus "persisting" (still enrolled) at the end of a six-year longitudinal study. I separate out the "persisting" group and, based on their transcript records, estimate the number who are "likely" to finish their degrees within an additional year. They are acknowledged in a separate variable.

Today, these lower-level indicators are certificates and licenses attached to degrees. Tomorrow, degrees may be replaced by certifications of proficiency across a broad array of knowledge and skill domains. The rapid rise of scalable, object-oriented curriculum (including the government’s SCORM model) will certainly make this desegregation easier and the needs of the marketplace will create a venue.

Can you comment on the idea that graduation rates will become increasingly marginalized with respect to their value as an indicator of a college’s performance as the idea of the ‘degree’ continues to be replaced by more granular indicators of competence? Can you also comment on the idea of a ‘degree’ as a certification of competence that can be more finely tuned to the learner’s needs than is possible with the current ideas of course structure, majors, departments, etc.?

Clifford Adelman:
    In 2000, the Department of Education published a study of mine entitled "A Parallel Postsecondary Universe: the Certification System in Information Technology" that explored some aspects of the questions you raise.

There is no question that the knowledge economy has created competency-based systems of certification that lie outside the formal higher-education universe. This phenomenon is global, and the most recent estimates are that, in IT alone, more than 2.5 million people have earned at least one certification (whether from vendors such as Microsoft or international "guilds" such as the Disaster Recovery Institute or from associations such as the National Association of Communications Systems Engineers).

What we knew of the domestic population that earned such certifications in IT was that 60-70 percent of them already held at least a bachelor's degree. Does that "marginalize" graduation rates? Not really, but it places the whole notion of credentialling in a much broader framework than we usually think of it.

Can "certification of competence" transcend specific majors? Twenty to twenty-five years ago, there were some experimental examinations of "academic competence" that were not subject-specific, but rather tried to tease out reliable measures of modes of thought: analysis, synthesis, differential perspective. These exams are fascinating to look at today in terms of their place in the evolution of assessment in American higher education. They didn't get anywhere for psychometric reasons (you don't want a long story here).

What you can find out is that student X, who is no longer at your institution, is now enrolled at Old Siwash. Maybe some day the system will also be able to tell you whether student X ever earned a degree from Old Siwash. But at present, our tracking ability is limited.

The question to be addressed is how to track an individual from day one of college attendance through degree completion even if he or she attends several institutions. One comprehensive database on degree completion that crosses institutions is the National Student Clearinghouse. I believe that at least 80% of universities use the Clearinghouse to assist students (or employers) who need to verify degrees. Has anyone discussed the possibility of using the data in the Clearinghouse to track individual students who move from institution to institution?

Clifford Adelman:
    First, as I recall, institutional membership in the Clearinghouse is voluntary (and there are fees) and we don't have 100 percent coverage. The current use of the Clearinghouse (as I recall) is confined to locating students who left your school and may be somewhere else, but cannot confirm graduation. As the American Council on Education's Terry Hartle said in the Chronicle article, a national tracking system would be very complicated, indeed.

I hope the discussions about student performance will take some of this more detailed transcript-based information on types of remediation into account.

Do you think these goals address the problems with graduation rates as a measure that your study highlights?

Clifford Adelman:
    Your first recommendation is really fascinating, if I hear it correctly. Does it mean that if a student leaves a school before earning a degree it is the responsibility of that school to help the student enroll somewhere else? And what would the time limitation be for enrolling somewhere else? For the second question, I would recommend (based on research with "event cohort" longitudinal studies) that the institutions discriminate by the AGE of the student at the point of entry. "Older" (you decide what that means) students are on complete different life trajectories than those who enter, let us say, by age 20. Your institutional research folks will no doubt report very different parameters for time-to-degree.

The present formula for calculating graduation rates does not allow including students who have completed the required number of credit hours for graduation but never bother to graduate. This is particularly true for community colleges. Doesn't it make sense to count this group of students in the success column? Can you comment on this?

Clifford Adelman:
    There is a section of the report that describes "associate's degree eligible-but no degree." A couple of years ago, we did a panel on this at the American Association of Community Colleges' annual convention. Contact Bob Garber, Vice Chancellor at San Diego Mesa College, to see what one institution did to improve its degree completion rates by tracking these students down.