Graduate-level programs were once relatively immune from pressure to define and measure “learning outcomes” for their students. But for good or ill, the student-learning-assessment movement has begun to migrate from the undergraduate world into master’s and doctoral programs. (At some institutions, there is even talk of defining a set of “foundational outcomes” for all graduate students—that is, a set of learning goals that would be analogous to general-education goals for undergraduates.)
On Wednesday morning, as the annual meeting of the Council of Graduate Schools got under way in Washington, three graduate deans led a workshop on assessing graduate students’ learning and using such assessments to improve programs.
Formal assessment for improvement, they said, is more useful and less painful than many faculty members believe. (And in any case, accreditors are insisting on it.)
The three deans sat down for an interview after the workshop.
Q. In doctoral programs with intense mentor-apprentice relationships, the idea of establishing rubrics and other lists of learning outcomes might seem off-key. If I’m a senior professor of comparative literature and I’ve supervised 30 dissertations during my career, I probably know in my bones what successful learning in my program looks like. Why should I be asked to write out point-by-point lists of the skills and learning outcomes that my students should possess?
Charles Caramello, associate provost for academic affairs and dean of the graduate school at the University of Maryland: If you write out lists of learning outcomes, you’re making the invisible visible. That’s really my answer. We’ve all internalized these standards. They’re largely invisible to us. Assessment brings them out into visibility, and therefore gives them a history.
William R. Wiener, vice provost for research and dean of the graduate school at Marquette University, who is currently dean in residence at the Council of Graduate Schools: There’s no way to aggregate and to learn unless you’ve got some common instruments. By having common instruments, we can see patterns that we couldn’t see before.
James C. Wimbush, dean of the University Graduate School at Indiana University: Part of the story has to do with the external enviroment. Because of the decrease in funding for state institutions, because of political pressures from state legislators, we are forced to be much more accountable. Our boards of trustees now are looking for more accountability. They don’t necessarily say, “We want to make sure that you’re doing assessments of graduate programs.” But they’re questioning, Do we have too many graduate programs? We have to do a better job of being accountable for how we use our resources from the state and elsewhere. Assessment is one of the ways of doing that.
William Wiener: And not only at public institutions. My Board of Directors asks the same questions.
Charles Caramello: Faculty care about standards. They really care about excellence. They really care about evaluation, and they really care about peer review. To the extent that you can say, Look, assessment is a form of all of these things—it’s not alien to what you do every day. It’s another name for it, and a slightly different way of doing it. And the great advantage of it is that it gives you a way to aggregate information, and therefore to see patterns.
Q. What about graduate programs that are now being asked to do student-learning assessments for two accrediting bodies? An engineering program, for example, might now be expected to do student-learning evaluation both for the specialized engineering accreditor and for its university’s regional accreditor.
James Wimbush: Yes, that happens. The school of education, the school of business—they have very rigid accreditation standards from their associations. They tend to focus on meeting those particular criteria.
Charles Caramello: But those programs tend to come on board most quickly with student-learning assessment because for them this is familiar. One important thing that we try to do at Maryland is not ask these programs to do the same thing twice. If they’re already using an assessment model for their specialized accreditor, we don’t want to tell them that they have to create a second model. We’ll find a way to work with them.
William Wiener: But sometimes there are elements that are missing. The outside accreditors are concerned with their own standards. They’re not always so concerned with the mission of the university.
Q. Once a university has developed learning goals for its graduate programs and has been through several cycles of assessment, how public do you want to make that information?
William Wiener: I think it should be public. I think it will give our public confidence in what we’re doing. I think the universities are afraid. But I think that will change. Where a program is low, so be it. What’s important is, Do they improve over time? And if you don’t start with something, you’re not going to go to the next level.
Charles Caramello: Programs are wary, and with some reason. You can’t create a situation where a program is shamed. Publicly, the message to put forward is, This is what we’ve discovered, and this is what we’re doing to improve. That’s useful to students, it’s useful to prospective students, it’s useful to the faculty in the program, it’s useful to the dean and provost. And that’s a real form of accountability. That’s not numbers. It’s “We found this problem. We’re going to fix this problem.” And then you can look two or three or five years later and see. Has the problem been fixed?
Experts explore the quality and assessment of higher education.
23 Responses to Measurement of ‘Learning Outcomes’ Comes to Graduate School
cirencester - December 2, 2010 at 6:31 am
The idea is sound, but I suggest that the assessment should be of knowledge or capability rather than learning in a specific program. An employer will be interested in what the graduate can do, not simply what he may have learned during his years at Attaboy U.
Not too long ago, colleagues and I noted the recent proliferation of MBA Programs and the lack of accountability even in well-established programs, where students get by on group exercises et al in a financially-stressed environment, making the failure rate almost non-existent. We recognized that accreditation evaluates the input, but there was no independent evaluation of the output. In response, we created the Certified MBA Program for post-graduate administration to test the output. NOBODY — deans, employers, students, accreditors, et al bought into the idea, relying on “accreditation” as the quality standard, so it died. Deans actively resisted the ides — possibly fearing exposure of deficient programs? Soon after, the AACSB promoted its “Assessment of Learning” program, which appears to eschew commong standards and allows universities to develop their own learning goals, relying mostly on inputs still, without any objective assessment of the outputs. This leads to the New Order Business School, advocated by Seth Godin in Fast Company, which admits one out of 1000 applicants, who spend minimum time in concventional classes, but are warmly received by emloyers who recognize they were admitted to the most competitive program (a la Harvard Business School). Alas — just as with debits and credits and T-accounts, “we have alwsys done it this way”, so don’t upset things with more intuitive approaches.
(We even contemplated the “Certified MBA Equivalent” (CMBAE) to recognize “can do”, rather than “where been”).
cirencester - December 2, 2010 at 6:41 am
(Sorry about two typos above — “idea” and “conventional”)
crankycat - December 2, 2010 at 7:12 am
Don’t even go there. If they go there, I’m outta here.
quidditas - December 2, 2010 at 7:18 am
“I probably know in my bones what successful learning in my program looks like. Why should I be asked to write out point-by-point lists of the skills and learning outcomes that my students should possess?”
Because they’re going to yank your tenure and kick you to the curb if you don’t. Also, it would be nice if your students didn’t have to dissect your corpse in order to know what their curriculum was.
22228715 - December 2, 2010 at 7:42 am
Post from quiditas: “Because I said so, or else” is rarely a successful or lasting strategy at any level, with any group. But I guess you could try it.
On the one hand, converting what I “know in my bones” to course content and books is a pretty hefty percentage of what faculty are supposed to be doing. This process asks faculty to do for teaching process what they already do for discipline content.
On the other hand, converting such high-level skills and abilities and growth to point-by-point lists is daunting. How do you boil down such things? If the goal is to teach your child to ride a bike… OK, we can write those learning outcomes. But if the goal is to teach her how to have healthy adult relationships… hmmm…
jffoster - December 2, 2010 at 8:00 am
Dumb it all down, and let the educationists control it, so that grad school and grade school become indistinguishable.
thomasluxon - December 2, 2010 at 8:16 am
I do faculty development for teaching. From where I sit, the lousiest motivation for articulating learning outcomes and designing program-wide measures is accountability. The best motivation, in my opinion is the healthy environment such activity creates for professional development. We’re better situated to talk about teaching and share our expertise if we spend time articulating what we expect of students and explaining how we measure their success.
thomasluxon - December 2, 2010 at 8:17 am
I do faculty development for teaching. From where I sit, the lousiest motivation for articulating learning outcomes and designing program-wide measures is accountability. The best motivation, in my opinion is the healthy environment such activity creates for professional development. We’re better situated to talk about teaching and share our expertise if we spend time articulating what we expect of students and explaining how we measure their success.
Thomas H. Luxon, Dartmouth Center for the Advancement of Learning
tay192 - December 2, 2010 at 9:14 am
Ideally, thomasluxon’s description would be the outcome of outcome assessment; however, much of this at the undergraduate level is, frankly, ridiculous. Goals are set very low to avoid getting “burned” by higher-ups and often they simply relate to “skill-sets.” Many faculty consider it “busy work” that keeps the dean’s office happy. I agree that there should be a visible record of what is taught and what is learned, but does that record have to be a “list” of data points? Why not introduce graduate and Undergraduate disciplinary journals? Other venues for student scholarship? With these we could see the quality and relevance of a curriculum.
drmhp - December 2, 2010 at 9:36 am
The faculty in my Ph.D. program used “performance appraisals” – which in many ways were outcome assessments that were simply not reported to anyone outside the program. The faculty came together to define several indicators of success for the students that reflected the values of the program (e.g., collaborative research and scholarship, program citizenship, etc.) and students met with their adviser on an annual basis to review their progress. This process gave students and faculty the opportunity to sit down and discuss a students’ progress throughout their time in the program on a broad level (in contrast to the day-to-day advising that takes place surrounding thesis and dissertation work, other research projects, etc.). Had the faculty simply aggregated these performance appraisal ratings and reported them to an administrator (e.g., a Dean, etc.) they could have been called “learning outcomes.”
The fact of the matter is that the grading scale in graduate programs is typically something like (A = A, A- = B, B = C, and C = F) – so finding other quantifyable measures of what students are gaining from the program makes sense. I think faculty can find unique ways of doing this that can be beneficial for a variety of stakeholders in the graduate education process.
sm0832 - December 2, 2010 at 9:43 am
I am glad that education is being held to higher professional standards. I am a student who passed my orals and now have a federal case as a result of other professionals not agreeing with my committee approval of my dissertation. This has been a two year battle for me, and a emotional draining process.
mrmars - December 2, 2010 at 10:08 am
For the sciences at least, graduate-level “assessment” would seem to be unnecessary. To get into a science-oriented graduate program at a major university takes a track record, years of demonstrated success at the undergraduate level. The “connection” is this: undergraduate assessment efforts (IMHO) are little more than attempts to gloss over the all too obvious fact that when learning doesn’t occur it’s about 97.9% traceable to admitting too many “scholars” who have little ability and even less motivation to succeed. Colleges do this because they have no other way of staying solvent in today’s anti-intellectual environment where kids graduate from high school with sterling grades based on the amount of “extra credit” offered in lieu of actually learning something, or learning how to learn (you want to obsess about measuring learning, do it there, at the HS level, and many of the higher level issues go away – automatically), but I digress.
Undergraduate science curriculum rigor selects against those who can’t or won’t learn, not that we don’t graduate far too many kids with “C” averages at the undergraduate level – we do. We take their money, we wince as they limp through their undergraduate years punctuated with too many “re-takes” of classes that they’ve failed and wish them well as they go off to start careers in retail sales. They certainly won’t get into graduate or professional schools, but they WERE paying customers while they were here, and its both unkind and – for many – all but impossible to try to get them to face reality along the way, so the charade goes on (and the classes remain full which keeps administrators happy!).
For the students who have the requisite ability and motivation, things work out as they should (and as they always have, even in the days before “assessment”). Their obvious achievement gains them access to quality graduate and professional programs where their work ethic and experience allow most to successfully complete an advanced degree. Success at the graduate level in science demands obvious proof of achievement, for both continuing faculty and their students. You wouldn’t suggest that we “assess” whether players in the NFL really have the necessary football skills, given how the system works, the mere fact that they are there is proof enough.
Am, I suggesting that all science graduate programs give out degrees of equal worth and that everyone with a science M.S. or Ph.D. is equally well trained? Of course not. Even in departments with a long-standing tradition of excellence, the occasional person “sneaks’ through for political or humanitarian (?) reasons, but this is human nature at work and assessment rubrics aren’t going to make a dent in that. There are also personal conflicts between student and advisor that preclude or complicate success, but that’s another issue. Prospective graduate students should realize that they get what they “pay” for (metaphorically speaking) and that departments that anyone can get into aren’t likely to offer as much value. For the vast majority of large “name brand” graduate programs, the fact that they continue to thrive in todays super competitive environment is proof enough of their effectiveness. If you want to find a graduate education issue that does need to be “assessed” lets promote discussions about matching output to need, an issue that’s been avoided like a rotting carcass (sorry, I’m a biologist) for generations.
11325240 - December 2, 2010 at 10:39 am
I work with faculty across the institution on assessment, and I entirely agree that accountability is the worst of reasons (least motivational, most likely to result in superficial rather than thoughtful assessment activities, most likely to end up as “busywork” done to satisfy outsiders rather than meaningful to those doing it) to do assessment. Paying attention to teaching & learning is the best of reasons. (I am not denying the genuine need to be accountable, but I still believe that is the “worst” rather than the best reason for devoting time to assessment.)
Faculty not yet involved in assessment often claim that it is poorly done and thus meaningless, producing sham positive findings. Any activity can be well or poorly done: research (even if published), service, teaching, test-writing, advising, mentoring, etc. So the fact that assessment can be poorly done is certainly not a reason to avoid it. The first classes any of us taught were almost certainly poorly done, but we stayed at it — and presumably got better at teaching over time. Your first forays into assessment will likely be disappointing too. But if you stay with it, you will find, a semester or two or three down the road, that you are increasingly focusing your courses on what students are learning rather than what you’re doing. That in itself is a hard shift — not surprising, since it’s obviously easier to focus on your own experience than that of others. And you will begin to find occasional surprises. Students will not be learning something you really thought they were. Students will be unexpectedly good at something you didn’t realize they could do. Those surprises become the reward of assessment. It turns out that what you were pretty sure about based on your experiences (or what you had never had a reason to wonder about) is not functioning quite as expected. And once that starts occurring, assessment becomes way more engaging. It turns out to be a lot of fun to engage in (informed — you now have data) conversations with colleagues about student learning.
That’s assessment — as it can work when done thoughtfully.
goodeyes - December 2, 2010 at 10:40 am
Can you imagine a student saying that taking the test on a subject was a waste of time because “I know in my bones” all the knowlege. Weak assessment means weak curriculum. It means that we just threw courses together and hope students get it. It means we have limited knowledge that students actually obtain the learning goals of our programs.
bondage2 - December 2, 2010 at 12:04 pm
I teach graduate Creative Writing. I assess my students constantly. I write four thousand word critiques of each student’s work three times each quarter. I give them high grades so that the can keep their fellowships, but meet with each of them individually several times per term for a more frank and realistic exchange of views-— not just evaluations, but conversations about their goals and work habits. Typically, it will take four or five years for any of this instruction, and even their harder and most diligent effort to result in some species of success as it is defined in my field. And the reality is, most of my students will never be successful in that sense. They may find teaching jobs and have a few good stories published, but in terms of art, they will hardly be great successes. So how do we assess what I do, what my Program does, what the students do as students, what they do as graduates? I suspect thevonly ones who worry about this are administrators. Those involved know the answers.
archman - December 2, 2010 at 1:37 pm
All this assessment junk would likely be pointless if we didn’t have so many open-enrollment graduate programs opening up right and left all over the place.
Now that the workforce is starting to see the “bad fruit” coming out of such programs, there is now all this concern for “accountability”. Hmmm…
nebo113 - December 2, 2010 at 2:21 pm
I’ll do it when physicians do it–with transparency!
kathden - December 2, 2010 at 2:35 pm
Why are there so many comments about “knowing it in my bones”? The guy asking the questions (David Glenn, I presume) put the phrase in the mouth of a fictional 30-year professor of comp lit, a figure he created for rhetorical purposes.
Assessment is being driven by those who don’t know but have power/money wanting sticks to beat teachers with. I agree that assessment can be useful when it is done for reasons intrinsic to learning and the processes of learning. In most cases at the graduate and undergraduate levels it can be done well simply by doing a comprehensive and comparative review of current students and then afterward asking whether you are achieving what you want. You can even put a lot of this in writing. But then those with power/money want what you’ve written to be rules and they want to quantify them. Thus bad assessment drives out good assessment.
davi2665 - December 2, 2010 at 3:29 pm
Establishing specific outcomes for graduate programs at least can help to shape a curriculum. I participated in the formation of a biomedical science program at a top 20 medical school for which a majority of the faculty argued that there should be no REQUIRED courses, no outcomes expectations other than publication of dissertation research, and individualized programs of study for each student. For the past 20 years, I have dealt with the consequences of such programs and experienced the frustration of directing basic sciences medical school and graduate courses for which participating faculty are supposed to be experts, since they possess a Ph.D. in that discipline. Yet, their knowledge is 1 cm wide and 50 km deep; most of them are utterly incapable of teaching even introductory lectures in the introductory course, even if it is in the subject area in which they do research. Where is the breadth of knowledge and the ability to provide a compelling overview? Often non-existent. That is why I only wanted M.D.-Ph.D. students carrying out research in my laboratory-
at least they knew the rudimentary knowledge of molecular biology, cell biology, and systems biology/physiology. Most universities have turned their Ph.D. graduate programs into a feeder system of cheap labor for faculty striving to get NIH grants and sustain their faculty positions. Unfortunately, the ultimate products of such graduate education are not prepared for either an academic position or a position in industry. Many of these graduate programs should be significantly reduced in size with an emphasis on the remaining programs and students receiving both a breadth and depth of training, not just a survey course or two, followed by a few lab rotations and a dissertation project.
rpm13 - December 2, 2010 at 11:56 pm
@drmhp: “The faculty in my Ph.D. program… define(d) several indicators of success…and students met with their adviser on an annual basis to review their progress…Had the faculty simply aggregated these performance appraisal ratings…they could have been called ‘learning outcomes.’”
Well, no. I did almost exactly what you suggest when I directed a doctoral program. When it only took me 10 minutes to respond to the request to enumerate the program’s learning outcomes, I was told I must not be doing it right. I had simply taken the graduate student manual and pasted in the list of accomplishments that students were expected to report every year. I even used proper assessment language: all the criteria were measurable behaviors. There are two problems with this. First, it’s the way things had always been done. Second it’s not time-consuming.
The emperor still has no clothes. Where are the data that show that formalized learning-outcomes-exercises are better than standard practice? Is the Assessment Industry accountable?
arrive2__net - December 3, 2010 at 4:06 am
A lot of grad school work is aimed at enabling the student to do original work, so it does not seem that a standardized assessment would really apply to a large part of what is learned. Perhaps you could define a core of knowledge and abilities that you could test, but much of what is learned by individual students is likely to fall outside the core.
To some extent grad school is often intended to prepare student to perform feats that may require months or years to complete, which perhaps implies that it would take months or years to assess it, in other words it would take a graduate school education to assess it.
Are course exams, papers, comprehensives, dissertations, theses, etc really of such little assessment value, or so incoherent that you can’t tell if the student is worthy? If so perhaps grad school should be more like basic training where everyone learns the same skill and passes the same test.
Bernard Schuster
Arrive2.net
drj50 - December 3, 2010 at 2:15 pm
Assessing graduate programs needn’t be difficult — and it’s not recent. Perhaps 15 years ago I heard a presentation on assessing graduate programs. Departments had very simple rubrics for evaluating theses and thesis defenses: research question, literature survey, methodology, results, conclusions, ability to talk about the above in the defense. Over time they were able get a picture of the things that their students did well (e.g., conclusions) and less well (e.g., literature review) and take steps to improve student performance.
I can imagine taking a few minutes after scoring comprehensive or qualifying exams to tabulate the results, which would identify areas in which students appear to be struggling . . . which could then be more intentionally addressed with the next generation(s) of students.
If we rely on data in our professional work in our disciplines, why are we so easily satisfied with anecdotal evidence, hunches, and “what we all know” in our pedagogy?
ollgu - December 3, 2010 at 4:31 pm
Articulating learning outcomes is, as someone suggested, beneficial because it makes the implicit explicit. One of the difficulties that few want to address in our approach to this are the methodologies used for accomplishing it. The assumption that quantitative results stemming from papers, exams, presentations, etc. are what drive most faculty to make adaptations to affect student learning is out of touch with what most faculty do – must do – to assist their students. Another issue that remains poorly addressed is how students are not included in the “measurement” process as a source of variability in student performance. Its an outcomes only methodology where process and student inputs affecting the process are largely left out. Many of the factors in student performance are never considered. Consequently, faculty are held responsibile for affecting student performance based on a behaviorist approach wherein faculty inputs and student outputs are given priority. Outside of the circle of practitioners who use and accept this methodological approach one would be hard pressed to gain acceptance of this from reviewers of research in the social/behavioral sciences. Again, I applaud the effort to make the intended goals of education more explicit. What I question are the methods used to do this. We need another (better) model.