As promised, this blog will expire with the calendar year (cue violins).

In this last post, I’ve tried to distill some fundamental arguments about assessment and accountability in higher education. I’ve borrowed liberally from comments that readers have left here and elsewhere on The Chronicle’s site. Many thanks to all of you for reading and arguing.

Of course there are more than two sides to these debates. In that respect, what follows is pathetically reductive. But I’ve tried not to put my thumb on the scale on behalf of either of these characters. I’ve tried to convey the strongest cases on each side of an admittedly-crudely-drawn line. (If I’ve failed to do that, you should of course call me out in the comments.)

The scene: Friday, 7:45 p.m. A bar on the outskirts of a moderately selective public university. The décor and the jukebox appeal to disillusioned 36-year-old faculty members and a few graduate students. The fraternities leave this place alone.

Accountability Skeptic: Did you see that memo from the dean today? He’s hired some consultant to teach us how to “design learning outcomes” for our students. I can’t imagine a bigger waste of time and money. And I don’t think the dean even believes in this stuff himself. I think he’s just trying to keep the accreditors off his back.

Accountability Hawk: Don’t be so cynical. Tuition and fees here have gone up by more than 50 percent since 2000. Students are taking on miserable levels of debt to be in our classrooms. They deserve to have faculty members who are focused on their learning—and that means that we need some kind of common understandings in our departments about the knowledge and skills students are supposed to be picking up.

Skeptic: Listen. I do focus on my students. I assess their learning every week. It’s called grading. I get the feeling that the dean wants a list of “learning outcomes” that are so easy or so nebulous that no student will really fail. It’s a joke.

Hawk: That might be true in some places. But some departments have developed very specific, very concrete rubrics that students sure as hell can fail. And when you talk about grades—I know that you take your grading seriously. But you and I both know that grade inflation is a serious problem on this campus. No one fully trusts grades anymore—not students, not accreditors, not employers.

Skeptic: Those are discussions that we should have inside our own departments. If we want to tackle grade inflation, we can do that by ourselves as a faculty. I don’t see what an outside consultant is going to bring to the table other than a lot of jargon. If I’m going to be forced to listen to a consultant lecture about “readiness assurance processes,” I’m going to need a stiff drink before I walk in the room. All of this is just going to add up to a lot of tedious, circular conversations about whether we’re ready to do assessment.

Hawk: What makes you think that we’re going to tackle grade inflation as a faculty? The problem has been growing for decades, and I haven’t seen any faculty uprising to fix it. You know what the incentives are like here. Untenured instructors are paranoid about their students’ course-evaluation scores, so they pump up their grades to keep the students happy. And people on the tenure track feel more pressure to publish research than to excel at teaching. If we start to talk seriously about learning outcomes, that’s our best chance to turn those incentives around. I’m glad the dean is finally paying attention to this.

Skeptic: Don’t you see how this all could backfire? There are a lot of reasons why universities sometimes promote research at the expense of teaching. One of those reasons is that department chairs and deans find research simple to quantify. When someone comes up for tenure, you can say, Ah, she has five publications that have been cited 16 times in all. It’s very simple to compare across faculty members. Never mind that there are serious problems with interpreting research-productivity numbers. What I’m afraid is that this learning-assessment industry is going to lead to some new kind of easily quantified but essentially bogus measure of faculty members’ performance as teachers.

Hawk: I think you’re letting the perfect be the enemy of the good here. Everyone knows that there are problems with counting publications and citations. But would you want to have a world with no information at all about how productive a faculty member is? It’s the same with measuring student learning outcomes. The tools we’re building will be imperfect, and everyone will know that, but it’s better than flying blind.

Skeptic: Yes, but, again—these are conversations we can have inside our departments and inside our disciplinary associations. I don’t trust the judgment of the “vice provost for assessment” floating around in the upper echelon of our administration. I don’t trust the assessment bureaucrats who work for our regional accreditor. And I don’t trust state or federal regulators. In the grand scheme of things, our university is basically fine. Our six-year graduation rate is 70 percent. Every hour that accreditors and regulators spend hassling us about “learning outcomes” is an hour when they’re not going after the real malefactors: diploma mills and colleges with graduation rates below 20 percent. And the same thing applies at the campus level: Every hour that our deans and provosts spend hassling the faculty about learning assessments is an hour when they’re not going after the handful of instructors who are truly incompetent and irresponsible.

Hawk: Do you really think our campus is basically fine? Really? Hundreds of students drop out here every year, and they mostly walk away with huge debts. A lot of others take seven years to graduate when they really ought to be able to finish in four. There are lots of reasons to believe that colleges like ours could have stronger graduation rates and could do better at making sure students learn actual skills while they’re here. For a start, we could make our credit-transfer policies simpler and more transparent. People in the accreditation agencies are more sophisticated about measuring student learning than you give them credit for. They understand how complicated all this is.

Skeptic: I get nervous these days when I hear people talk about how sophisticated they are at dealing with complicated academic data. Remember the National Research Council’s ratings of doctoral programs? Their statistical model was sophisticated, all right—so sophisticated that almost no one could understand the ratings. And the way they tallied research productivity turned out to be a serious problem for disciplines like political science and sociology. Did you see that letter to the TLS from Krishan Kumar? Or look at the Collegiate Learning Assessment. Colleges that use that test receive thorough, sophisticated reports that explain how their students compare to similar students at other colleges. But it seems like those sophisticated reports might be a little too sophisticated for some purposes. When colleges disclose their CLA scores to the Voluntary System of Accountability, the colleges often plug in the wrong language from their CLA reports—declaring that their CLA score gains are “below expected” when they’re actually “above expected,” and vice versa. That means that prospective students who visit the Voluntary System’s Web site are getting bad information.

Hawk: But that’s all just nitpicking, isn’t it? These systems aren’t perfect, but they’ll get better over time. The basic thing that we have to do is to disrupt the “non-aggression pact” between students and faculty members. Instructors need to press students to learn, and students need to demand serious attention from their instructors. I don’t think that’s going to happen without outside pressure. You can daydream about “conversations inside our department,” but everything in my experience tells me that those conversations just aren’t going to happen on their own. If we need pressure from a dean hassling us to define our learning outcomes, then so be it. I think this is actually an exciting time to be in the academy. Every week I read about some interesting new way to assess student performance.

Skeptic: Me, too.

(Photo by the Flickr user D. Sharon Pruitt. Used under a Creative Commons license.)

Matthew Denhart, administrative director at the Center for College Affordability and Productivity, an organization that advocates for greater transparency and accountability about student learning in higher education, said the report was “kind of refreshing,” even as he noted the inherent limitations of a report in which the university itself chooses what information it will publish.

He said he especially liked the data Phoenix collected on how students’ salaries (most of them work while attending) rose at a rate higher than the national average while they were enrolled. Data like that are “something you really have to struggle to find anywhere else,” Mr. Denhart said.

The findings for the 2009 academic year did show some warts—most notably, declines from 2008 in program-completion rates. In 2009, the proportion of Phoenix students completing an associate degree within three years of enrolling was 23 percent, down from 26 percent the year before. Among bachelor’s degree students, the six-year completion rate was 34 percent, versus 36 percent the previous year. (You can dig into the numbers from this year’s report, and the two previous ones, here. Read The Chronicle’s coverage of the previous reports here and here.)

University officials said they believed the “current economic conditions” might have contributed to the declines in graduation rates. Many Phoenix students may have faced financial hardships that caused them to interrupt their studies, the officials said—the same explanation many four-year nonprofit colleges recently gave in a recent Chronicle analysis of graduation rates.

In November, Phoenix instituted a free three-week orientation program that is mandatory for all students entering with fewer than 24 credits. In a test of the orientation involving 30,000 students, the university found that 80 percent of the students who started the program completed it, and that retention rates among those students was higher than for those who didn’t take it. Phoenix’s president, William J. Pepicello, said the university hopes the orientation will eventually lead to better completion rates. (Phoenix calculates a completion rate because so few of its students are the first-time freshmen typically counted in the standard “graduation rate” calculations.)

The Phoenix academic report also includes findings on students’ performance relative to hundreds of thousands of students at nearly 400 peer institutions on two standardized tests: the Standardized Assessment of Information Literacy Skills and the ETS Proficiency Profile (formerly called the Measure of Academic Proficiency and Progress).

For instance, on the Proficiency Profile, 2,428 University of Phoenix seniors slightly underperformed a comparison group of 42,649 seniors at peer institutions in critical thinking, humanities, social sciences, and natural sciences, and moderately underperformed the peer group in reading, writing, and mathematics. In comparisons of seniors versus freshmen within the university, the 2,428 seniors slightly outperformed 4,003 freshmen in all categories except natural sciences, in which they were equivalent.

Today The Chronicle published an analysis of recent changes in the six-year graduation rates at nearly 1,400 colleges. At most institutions, the rate ticked up at least modestly between 2003 and 2008. But at 35 percent of the colleges in the data set, the rate declined, in some cases steeply.

In other words, despite all the attention thrown at graduation rates during the last 15 years, many colleges’ numbers remain stagnant or worse.

But what exactly is a six-year graduation rate? Here are a few basics. (If you already live and breathe this stuff, this post isn’t for you. Go watch this instead.)

Q. What do these numbers represent?

A. In 1990, Congress passed the Student Right-to-Know Act, which requires colleges to disclose information on graduation rates and serious crimes.

In particular, the law requires colleges to report the proportion of students “completing their program within 150 percent of the normal time to completion.” For four-year colleges, that means the proportion of students who earn bachelor’s degrees within six years. In 1997 the federal government began to systematically collect those numbers through its Integrated Postsecondary Education Data System, commonly known as IPEDS.

Q. In general, how are colleges doing on that measure?

A. A little more than half of students who enter four-year colleges earn a bachelor’s degree from the same institution within six years. You tell me whether that’s good or bad.

Q. And how have the numbers changed since 1997?

A. Recent trends have been at least modestly positive. Among students who entered four-year colleges in 1996, 55.4 percent had earned bachelor’s degrees six years later. For the cohort of students who started college in 2001, the figure was 57.3 percent.  That’s according to data published by the federal government last year. If you study that table, you’ll see similar small, steady increases in most categories.

Q. Why does the IPEDS six-year graduation statistic make some people rend their garments and gnash their teeth?

A. It doesn’t cover people who begin college as part-time students. It doesn’t cover people who begin at community colleges and then transfer to four-year institutions. In fact, it doesn’t cover people who transfer at all: To get picked up in the federal data, students have to begin and end at the same institution. It doesn’t cover the nontrivial number of students who complete college seven or more years after they start. Whole swaths of higher education are rendered invisible.

Q. What could we use instead?

A. In many people’s eyes, the gold standard would be a unit-record tracking system that would follow students from institution to institution for the full length of their college careers. But Congress rejected that idea several years ago amid heavy political opposition.

The unit-record concept has not died, however. The Obama administration has given tens of millions of dollars to states to build data systems that would track students’ progress from elementary school through college. (The State Higher Education Executive Officers has recently published two reports about the best ways to create those databases.) Even if no true federal data system emerges, there will probably be a de facto national unit-record database within a decade or so.

Q. If we used a unit-record data system, would four-year colleges’ graduation rates look healthier than they do now under the IPEDS six-year graduation statistic?

A. It depends on what you want to measure. If four-year colleges received credit for graduating students who transferred in from community colleges, then their numbers would certainly look better. That’s an important topic, and most people agree that four-year colleges should be credited for playing that role.

But if we want to focus on students who begin their college careers at four-year colleges, then a unit-record system would probably make the national graduation rate look only somewhat better.

Unlike the IPEDS data, a unit-record system would capture students who begin at one four-year college and graduate from another four-year college. But including those transfer students would probably improve the national six-year graduation rate by only a few percentage points.

To see what I mean, look at the data released last week from the Beginning Postsecondary survey, a periodic federal study that tracks a sample of students through their college experiences. Among students who enrolled in four-year degree programs in 2003-4, 63.2 percent had earned a bachelor’s degree within six years. That’s better than the 57.3 percent rate I cited above from the national IPEDS data, but it’s not an enormous difference. (For more analysis of the Beginning Postsecondary data, see Kevin Carey’s Brainstorm post from last week.)

Q. Until the dawning of a unit-record-tracking age, the IPEDS six-year graduation rate will probably be the best available measure. Since we have to live with this system, are there ways we could improve it?

A. Maybe. In July, the National Postsecondary Education Cooperative published a white paper about how to improve the IPEDS graduation-rate calculations. Among other things, the report suggested that the federal government’s College Navigator Web Site should display five-year rolling averages of an institution’s graduation rate, rather than focusing solely on a single cohort.

(Photo by the Flickr user reality-check. Used under a Creative Commons license.)

During a panel on Thursday morning at the annual meeting of the Council of Graduate Schools, two deans asked the same questions that were raised by Robin Wilson in The Chronicle this week: Do graduate schools keep adequate track of what happens to their alumni? If not, why not?

“At most graduate schools, there appears to be very little central data collection about career outcomes,” said Patricia G. Calarco, dean of the graduate division at the University of California at San Francisco. Individual departments and programs often maintain their own databases, she noted, but the quality of those databases is hit or miss.

Lynne M. Pepall, dean of the Graduate School of Arts and Sciences at Tufts University, asked the audience members (who were wielding classroom-style “clickers”) how many of their institutions kept systematic records of alumni outcomes. Of the 34 respondents, 20 said that no office at their university kept careful track. Not the dean’s office, not the alumni office, not the institutional-research office, not even individual departments.

That is a shame, the two deans said, because graduate schools would serve their students better if they had a fuller sense of the careers the students want and the careers they ultimately find.

Faculty members often don’t want to hear it, Ms. Calarco said, but a substantial proportion of graduate students don’t plan to work in traditional academic or research jobs. A survey of UCSF students in 2008 found that if they could choose only one career path, fully a third of the students wanted nonresearch jobs. Last year, the campus started an internship program that places graduate students in industry and government settings.

One difficult and sometimes painful challenge, the deans said, is to gather information about what happens over the long term to people who drop out of doctoral programs.

Of the students who withdraw from doctoral programs at Tufts within the first six years, Ms. Pepall said, roughly half walk away with master’s degrees, and many of those people seem to do well. But the university does not know much about the long-term fates of the other half, she said, and it has a responsibility to learn about them.

(Image by the Flickr user Mike Licht. Used under a Creative Commons license.)

On Wednesday morning, as the annual meeting of the Council of Graduate Schools got under way in Washington, three graduate deans led a workshop on assessing graduate students’ learning and using such assessments to improve programs.

Formal assessment for improvement, they said, is more useful and less painful than many faculty members believe. (And in any case, accreditors are insisting on it.)

The three deans sat down for an interview after the workshop.

Q. In doctoral programs with intense mentor-apprentice relationships, the idea of establishing rubrics and other lists of learning outcomes might seem off-key. If I’m a senior professor of comparative literature and I’ve supervised 30 dissertations during my career, I probably know in my bones what successful learning in my program looks like. Why should I be asked to write out point-by-point lists of the skills and learning outcomes that my students should possess?

Charles Caramello, associate provost for academic affairs and dean of the graduate school at the University of Maryland: If you write out lists of learning outcomes, you’re making the invisible visible. That’s really my answer. We’ve all internalized these standards. They’re largely invisible to us. Assessment brings them out into visibility, and therefore gives them a history.

William R. Wiener, vice provost for research and dean of the graduate school at Marquette University, who is currently dean in residence at the Council of Graduate Schools: There’s no way to aggregate and to learn unless you’ve got some common instruments. By having common instruments, we can see patterns that we couldn’t see before.

James C. Wimbush, dean of the University Graduate School at Indiana University: Part of the story has to do with the external enviroment. Because of the decrease in funding for state institutions, because of political pressures from state legislators, we are forced to be much more accountable. Our boards of trustees now are looking for more accountability. They don’t necessarily say, “We want to make sure that you’re doing assessments of graduate programs.” But they’re questioning, Do we have too many graduate programs? We have to do a better job of being accountable for how we use our resources from the state and elsewhere. Assessment is one of the ways of doing that.

William Wiener: And not only at public institutions. My Board of Directors asks the same questions.

Charles Caramello: Faculty care about standards. They really care about excellence. They really care about evaluation, and they really care about peer review. To the extent that you can say, Look, assessment is a form of all of these things—it’s not alien to what you do every day. It’s another name for it, and a slightly different way of doing it. And the great advantage of it is that it gives you a way to aggregate information, and therefore to see patterns.

Q. What about graduate programs that are now being asked to do student-learning assessments for two accrediting bodies? An engineering program, for example, might now be expected to do student-learning evaluation both for the specialized engineering accreditor and for its university’s regional accreditor.

James Wimbush: Yes, that happens. The school of education, the school of business—they have very rigid accreditation standards from their associations. They tend to focus on meeting those particular criteria.

Charles Caramello: But those programs tend to come on board most quickly with student-learning assessment because for them this is familiar. One important thing that we try to do at Maryland is not ask these programs to do the same thing twice. If they’re already using an assessment model for their specialized accreditor, we don’t want to tell them that they have to create a second model. We’ll find a way to work with them.

William Wiener: But sometimes there are elements that are missing. The outside accreditors are concerned with their own standards. They’re not always so concerned with the mission of the university.

Q. Once a university has developed learning goals for its graduate programs and has been through several cycles of assessment, how public do you want to make that information?

William Wiener: I think it should be public. I think it will give our public confidence in what we’re doing. I think the universities are afraid. But I think that will change. Where a program is low, so be it. What’s important is, Do they improve over time? And if you don’t start with something, you’re not going to go to the next level.

Charles Caramello: Programs are wary, and with some reason. You can’t create a situation where a program is shamed. Publicly, the message to put forward is, This is what we’ve discovered, and this is what we’re doing to improve. That’s useful to students, it’s useful to prospective students, it’s useful to the faculty in the program, it’s useful to the dean and provost. And that’s a real form of accountability. That’s not numbers. It’s “We found this problem. We’re going to fix this problem.” And then you can look two or three or five years later and see. Has the problem been fixed?

You don’t want that list to just sit there on a Web site as a testimony to your college’s good intentions. (Right?) You want to take reasonable steps to measure whether your students are actually meeting the goals you’ve defined.

How best to do that is, of course, a highly contested question.

Some scholars urge colleges to use nationally normed tests, like the Collegiate Learning Assessment, that attempt to capture students’ critical-thinking and analytic-writing skills. Others say it is better to use student portfolios that allow students to demonstrate their skills in the context of their course work. (For a taste of that debate, see this post and the comments it engendered.)

Charles Blaich, director of Wabash College’s Center of Inquiry in the Liberal Arts, advocates an all-of-the-above approach. Colleges should use as many reasonable kinds of data as they can get their hands on, he says. The CLA and other national tests can be powerful tools, but they can’t possibly capture a college’s full range of learning objectives.

Mr. Blaich’s center is leading a new study in which 30 colleges and universities will try to synthesize multiple kinds of student-learning data.

“To the greatest extent possible, we want to help institutions use data that they already have,” Mr. Blaich says. “We don’t want them to have to create elaborate new structures for collecting data.”

Participating colleges will be welcome to use scores from CLA-style tests, and they will also be encouraged to dig deeply into their institutional data from the National Survey of Student Engagement and its ilk. But most of all, they will be expected to use materials from student course work.

“Each institution will have to figure out how it wants to do that,” Mr. Blaich says. “But we want them use stuff that students actually produce, and to use that information for assessment and improvement. We see this as a more sustainable model for colleges, something that turns down the temperature on data collection.”

Over a three-to-four-year timetable, each college will focus on one or two specific learning outcomes and experiment with using student-outcome data to improve classroom instruction.

“The issue we see with institutions,” Mr. Blaich says, “is actually finding processes to use the data that they have. That’s the biggest challenge. In a way, we’ve already got a lot of the assessment things down. We’ve got rubrics. We’ve got e-portfolios. We have all sorts of stuff out there. But we need to improve its yield. And that’s more of a political process, more of a cultural process.”

The 30 participating institutions have a variety of plans for their Wabash-study grants. Middlebury College and Kalamazoo College plan to use the project to assess and improve their senior-year capstone experiences.

At St. Lawrence University, faculty members plan to use the Wabash project to find ways to improve students’ quantitative literacy, writing and research skills, and appreciation of diverse cultures. They will study not only what goes on in the classroom, but also the effects of students’ visits to the university’s Quantitative Resource Center and other student-support services.

“I think one of the aspects of Wabash that fits with the philosophy of St. Lawrence is that it emphases the importance of pedagogy,” says R. Danielle Egan, an associate professor of gender studies who is helping coordinate the project, in an e-mail message to The Chronicle. “That is not a hard sell here. It’s central to our identity as a university.”

Ms. Egan says she understands that some faculty members “cringe (or worse)” when assessment projects arise, but she hopes that a project like this one, which is grounded in students’ course work and not tied to national tests, will win broad acceptance.

She adds that she expects the project to give students “a more transparent understanding of not only what they have been doing but why they have been doing it over their four years at SLU. I think that making learning goals transparent should help students become more intentional—or at the very least provide clarity.”

Westminster College, in Utah, will use the project to focus on some of its collegewide learning goals:  “global consciousness, social responsibility, and ethical awareness.”

“That’s been the goal that we’ve wrestled with the most, as far as exactly what it entails,” says Paul K. Presson, Westminster’s associate provost for institutional research. “We’ve been trying to come up with meaningful and measurable ways of addressing consciousness and ethical awareness.”

That last comment brings us to one element of the widespread skepticism that learning-assessment projects face.

Are colleges trying to assess aspects of personal development—including student behavior outside the classroom—that really can’t or shouldn’t be measured? Is there something slightly creepy and hyperintrusive about some of this work?

“There are no chips in the neck here,” Mr. Blaich says. “I think what colleges are trying to do is to see to what extent the activities that students engage in—in terms of organizations, study abroad, and so on—to what extent are those improving things in terms of things like diversity outcomes? A liberal-arts education is meant to be a sort of seamless in-and-out-of-the-classroom environment, where all sorts of things that are going on may influence student learning. Colleges would like to get a sense of whether the activities they’re sponsoring outside of class are benefiting students as much as they hope.”

I was very pleased to see the responses to the announcement of the Presidents’ Alliance as generally welcoming (“commendable,” “laudatory initiative,” “applaud”) the shared commitment of these 71 founding institutions to do more—and do it publicly and cooperatively—with regard to gathering, reporting, and using evidence of student learning.

The set of comments is a fairly representative sample of positions on the issues of evidence, assessment, and accountability. We all agree that higher education needs to do more to develop evidence of student learning, to use it to measure and improve our work, and to be far more transparent and accountable in reporting the results. The comments suggest different approaches—and these differences are more complementary than contradictory—to where we should focus our efforts and how change will occur. I’d suggest that none of us has the answer, and while each of these approaches faces obstacles, each can contribute to progress in this work.

For William Chace, Cliff Adelman, and Michael Poliakoff, the focus should be on some overarching measures or concepts that will clearly tell us, our students, and the public how well we are doing. Obtaining agreement on a “scale and index,” or the appropriate “active verbs” describing competence, or dashboards and other common reporting mechanisms will drive change by establishing a common framework for evaluation.

Josipa Roksa, on the other hand, suggests that real change will only happen from the ground up. Faculty training, incentives, and classroom practices need to be strengthened; change will occur when “faculty are capable of and will teach specific skills that are identified as important and/or in need of improvement by assessment endeavors.” In both cases, the comments suggest a focus and view of change somewhat different from what they see in the Presidents’ Alliance.

As the Presidents’ Alliance moves forward, it will be important to keep these perspectives and points in mind. At the same time, these perspectives too may miss some important elements that are emphasized in the Presidents’ Alliance. 

First, too narrow a focus on either some overarching measures or conceptual formulations may end up with these being disconnected from improvement. For example, establishing institutional indicators of educational progress that could be valuable in increasing transparency may not suggest what needs changing to improve results. As Adelman’s implied critique of the CLA indicates, we may end up with an indicator without connections to practice. At the same time, Adelman’s move from “active verbs” to “sample assessments” and then presumably to improvement are not small and seamless steps. The Presidents’ Alliance’s focus on and encouragement of institutional efforts is important to making these connections and steps in a direct way supporting improvement.

Second, it is hard to disagree with the notion that ultimately evidence-based improvement will occur only if faculty members are appropriately trained and encouraged to improve their classroom work with undergraduates. Certainly there has to be some connection between and among various levels of assessment—classroom, program, department, and institution—in order to have evidence that serves both to aid improvement and to provide transparency and accountability. But these connections are unlikely to occur spontaneously. In enlisting presidents, our initiative is trying to encourage academic leaders to set an agenda that would include faculty as part of a broader institutional endeavor.

As Laura Perna’s note suggests, the Presidents’ Alliance is setting forth a common framework of “critical dimensions” that institutions can use to evaluate and extend their own efforts, efforts that would include better reporting for transparency and accountability and greater involvement of faculty. Equally important, she also notes that the public nature of these commitments and “a commitment to sharing information” hold the promise of forming a more coherent professional community around issues of evidence, assessment, and accountability.

As Perna’s comment also points out, there is wide variation in where institutions are in their efforts, and we have a long way to go. But what is critical here is the public commitment of these institutions to work on their campuses and together to improve the gathering and reporting of evidence of student learning and, in turn, using evidence to improve outcomes. The Presidents’ Alliance provides a shared framework for institutional self-evaluation and action that will encourage them in their efforts to fulfill these commitments.

We hope many other institutions will join the Presidents’ Alliance and make similar commitments. The involvement of institutions of all types will make it possible to build a more coherent and cohesive professional community in which evidence-based improvement of student learning is tangible, visible, and ongoing.

Michael Poliakoff, policy director at the American Council of Trustees and Alumni:

We applaud the desire of university presidents to improve student learning and accountability on their campuses, although the idea is hardly new. A number of institutions in the alliance, in fact, distinguished themselves by their work in assessment and accountability for student learning before the organization was formed.

Does progress in these areas require another coalition? The alliance offers a four-point plan for presidents to gather, use, report and publicize student learning outcomes. What’s been stopping all institutions from doing this long before now?

The availability of appropriate tools has not been an obstacle. Hundreds of institutions around the nation—indeed, many within the alliance—use effective and remarkably inexpensive instruments to evaluate student learning, such as the Collegiate Learning Assessment (CLA), the Measurement of Academic Proficiency and Progress (MAPP), and the Collegiate Assessment of Academic Proficiency (CAAP).

The CLA in particular has generated important findings about undergraduate cognitive growth in different academic programs, and is in particularly wide use among participants in the Association of Public and Land-Grant Universities’ Voluntary System of Accountability. The South Dakota Board of Regents for some years has used the CAAP to make general education assessment of students in its state colleges and universities a mandatory element in performance-based funding.

Rather than creating a new coalition and spending money on national conferences to celebrate their successes, wouldn’t it be better for campus leadership to use limited time and resources to further the hard but productive work of serious data collection and analysis? By serious data collection, we don’t mean portfolios—which, notably, is one of the options the alliance considers. Portfolios, although labor-intensive, offer at best supplementary material to understand an undergraduate’s development. At worst, they evade and distract from clear measures of academic progress. What higher-education leadership needs is objective assessment of academic value added, along with outcomes measures like licensure and professional examination results. It is well within reach.

Finally, it does not suffice to “[ensure] that at least once a year the governing board … receives and discusses a report on efforts,” as the outline states. Governing boards and the campus community need clear, reliable data that they can see in dashboard indicators as well as interpreted in a full report. Underlying any report on student learning must be accountability based on performance data. Press releases, national associations, and conferences don’t count.

Josipa Roksa, assistant professor of sociology at the U. of Virginia and co-author of Academically Adrift: Limited Learning on College Campuses (forthcoming in January from the U. of Chicago Press):

The Presidents’ Alliance for Excellence signifies the growing attention of higher-education leadership to student learning, which is a welcomed and much needed development. Alliance members should be applauded for their vision and commitment to place learning on institutional and national agendas.

While this laudatory initiative has great potential, some might worry about its conspicuous absences.  For example, our leading universities, the ones that excel at selecting and attracting the “best and brightest,” do not appear to be deeply invested in the endeavor.  According to the most recent U.S. News & World Report rankings (an admittedly imperfect and problematic metric), no institution among the top 20 national universities is participating in the alliance, and only two from the top 50 are involved (the University of Southern California and the University of Wisconsin at Madison).  The absence of the leading universities is notable not only because they teach undergraduates, but also because they are deemed to be the best places to train the future professoriate and often serve as organizational models for the larger field.

And there is another conspicuous absence: the faculty, or, more specifically, the classroom.

Among the four main categories of commitments, three focus on collecting and reporting data, while only one is about using data.  Moreover, none of the main categories or subcategories includes a reference to training current and future faculty (i.e., graduate students) in teaching and learning, changing institutional incentives to reward excellent teaching, building coalitions with and among faculty regarding teaching standards and practices, or the like.  Institutions often note that they will work with faculty and students to develop assessment strategies and program initiatives.  But the broadness and vagueness of the majority of the statements appear conducive more to simply adding the issue to already crowded institutional agendas, rather than to providing faculty with the tools and incentives to change what happens in curriculum and instruction.

Reading the plans, one gets too limited a sense of how presidents will ensure that their faculty are capable of and will teach specific skills in the classroom that are identified as important and/or in need of improvement by assessment endeavors.  The closest some plans seem to come is to propose specific courses, whether on writing, critical thinking, globalization, or whatever is seen as a weakness.  But a semester of exposure to complex ideas and processes is not sufficient to transform undergraduates’ learning experiences.

While this initiative is certainly welcome and should be embraced by all those concerned with improving undergraduate education, bolder and more far-reaching endeavors for changing classroom practices, and providing training and incentives for faculty, will hopefully emerge from the work of the alliance and others moving forward.  Problems with undergraduate education are significant and require more than a few adjustments to business as usual.

Laura W. Perna, professor of education at the U. of Pennsylvania and an editor of Theoretical Perspectives on Student Success: Understanding the Contributions of the Disciplines (Jossey-Bass, 2008):

The pledges announced by the Presidents’ Alliance for Excellence in Student Learning and Accountability are an important step toward providing the public with information about the quality of student learning at our nation’s colleges and universities.

The strategies and approaches outlined in the current and planned initiatives of the 71 participating institutions vary greatly.  Such variation is not only expected but also appropriate, given the great diversity of America’s colleges and universities and the importance of institutional autonomy. A review of current and planned institutional initiatives also shows that progress toward assessing student-learning outcomes varies greatly. Nonetheless, together the pledges offer hope for greater progress.

The pledges generally include attention to many critical dimensions of the process, including the specification of learning outcomes; the identification of approaches to assessing these outcomes; consideration of the most appropriate national and institutional sources of data for assessing outcomes and for the systems required to manage and analyze the data; procedures for passively and actively disseminating results; and approaches to involving and supporting faculty in the assessment of student learning outcomes.

Particularly promising is the way that these pledges reflect, and likely further encourage, a commitment to sharing information within and across institutions about how to best assess student-learning outcomes.

We’ve invited several people to take a look at those 71 pledges and to share their thoughts, and we’ll present their comments this week. In this first round, we’ve got William Chace and Cliff Adelman.

William M. Chace, president emeritus of Emory University and author of 100 Semesters (Princeton University Press, 2006):

This effort, which in part is meant to “gather evidence about how well students in various programs are achieving learning goals across the curriculum,” is wholly commendable. It is also wholly belated in its appearance.

For decades, the woeful reluctance of America’s universities and colleges to assess outcomes, to make clear to both students and parents what their investment in education has yielded, and to publicize nationally those results, has been embarrassing and destructive to the reputation of those schools. Higher education in this country has become stunningly expensive, yet it is something for which a great deal has been paid with no concrete evidence of its worth. For too long, those who pay have been asked to “trust” while the schools have not been required to “verify.”

Time will tell if the solemn dedication of the 71 presidents will produce anything substantial. Most of the presidential signers lead private liberal-arts colleges; few public institutions are represented; no Ivy League school has signed up; the effort, to succeed, will need the leading heft and thrust of the most prestigious schools.

Moreover, all the signatories will need to do what they now abjure: to adopt a national scale and index of educational success. While it is understandable that “each college and university … should develop ambitious, specific, and clearly stated goals for student learning appropriate to its mission, resources, tradition, student body, and community setting,” clinging to such autonomy will allow every school to declare success on its own terms. The powerful truths of competition—yes, some schools do a better job than others—will reveal, for parents and students, where best to place their tuition money, and this will only serve to enhance teaching and learning at all the schools.

Clifford Adelman, senior associate at the Institute for Higher Education Policy:

It’s all blah unless …

… the presidents—and their chief academic officers, deans, department chairs, faculty, and students—take something like the Degree Qualifications Profile to be released in draft form for a two-to-three-year national discussion by the Lumina Foundation in January, and (a) work through every proposed competence for associate, bachelor’s, and master’s degrees; (b) add or modify using the same genre of active verbs that mark what students qualifying for these degrees know and can do; and (c) provide samples of assessments used in their programs that validate and benchmark student competencies that flow from those active verbs.

If all we get are cliches and non-operational terms such as “critical thinking,” “problem solving,” “appreciation,” “awareness,” and “ability,” or if all we do is administer a test to a sample of paid volunteers, regress their scores on our beloved SAT, and produce effect size changes, we get nothing. We need the transformational approach to student learning outcomes that the Degree Qualifications Profile challenges us with, and both the curricular and delivery modifications that will inevitably result from rising to those challenges. If presidents and CAO’s don’t engage deeply with this proposition, we’ll be lost in a sea of blah.