In his commentary the other day on the National Research Council’s assessments of doctoral programs, Jason Thomas Parker warned, “By outsourcing evaluation of our doctoral programs to an external agency, we allow ourselves to play the double game of insulating ourselves from the criticisms they may raise by questioning their accuracy, while embracing the praise they bestow.”
One program that has embraced the NRC’s praise is Ohio State University’s School of Communication, whose doctoral program received some impressive numbers in the new report. In an announcement last week, the program trumpeted its faculty’s “absolute rank of #1 in research activity” and the program’s overall “#3 ranking in the field.”
All of that may be true, and it may be well earned. But take note: Ohio State’s School of Communication is (almost literally) the poster child for potential inaccuracies in the NRC report.
Back in June, The Chronicle published an article about scholars’ concerns that the NRC’s data might be growing stale. The surveys that underlie the report were conducted back in 2006, and most of the data concern the 2005-6 academic year. Quite a few departments have seen heavy faculty turnover in the last several years.
To illustrate that point, we described the huge expansion of Ohio State’s communications program since 2006-7. Scroll to the bottom of that article, and you’ll see a graphic that tells the tale. Of the 18 faculty members in the program in 2006-7, four have left. Meanwhile, 19 new faculty members have been hired. So the new NRC report is based on data that apply to fewer than half of the program’s present-day faculty members.
None of this is meant to disparage the Ohio State communications program. It may indeed be one of the best in the country, both in 2006 and today. Its high rankings and its ability to lure so many new faculty members are presumably signs that it is doing something right. And the department did not approach The Chronicle last spring to point out its faculty turnover (we discovered that on our own), so its self-congratulatory announcement isn’t guilty of the kind of hypocrisy that Jason Thomas Parker criticized.
But still, it’s hard not to wonder: If the department’s rankings hadn’t looked so good last week, would it have posted a note about how much its faculty had changed since 2006, thereby raising questions about the accuracy of the entire NRC report?
Experts explore the quality and assessment of higher education.
7 Responses to Exhibit A in the NRC Rankings’ Problems
kfroilan - October 5, 2010 at 3:57 pm
For those who may not click over to read the entire OSU announcement, they do acknowledge that the data used for the report only goes through the 2005-2006 school year. While the announcement may be self-congratulatory (and what department wouldn’t jump on a ranking statistic like that?), it is also not trying to mislead anyone on the age of the data.
osucomm - October 6, 2010 at 9:26 am
If you consider the direction of change in the OSU program, it would be most likely that OSU’s ranking would increase rather than decrease based on the hiring of an endowed professor and faculty from other top PhD programs during this time frame. Maybe most importantly, the OSU Communication program has consistently been ranked among the top PhD programs in the field by a number of sources independent of the NRC in recent years, including being identified as one of the top three most prolific communication faculty in the country in terms of peer-reviewed publications by Science Watch (http://sciencewatch.com/dr/sci/08/aug31-08_1/) and, using the most up-to-date data, top in publication in various subdisciplines in the field by ComAnalytics (http://www.cios.org/comanalytics).So, the factual point of the article that the data are out of date is well-taken. But, given the corroborating evidence and often more recent data indicating OSU’s continued high ranking — and the likely direction of change in quality due to faculty replacement and additions — I would think it may have been better to select a “poster child” program ranked in the top that had recently lost top faculty and replaced them with less productive faculty rather than OSU which is clearly on an upward trajectory.
brite - October 6, 2010 at 12:39 pm
I would just like to point out regarding the NRC rankings, as a doctoral student myself, that the “lagging” statistics in fact assess departmental culture for doctoral students who are just now reaching the job market. While there are indeed numerous problems with citing data from 2006-7 to bolster departmental reputations, they at least are a good metric for the quality of experiences of recent Ph.Ds. and perhaps a worthwhile guide for evaluating candidates over the next few years.
leapter - October 6, 2010 at 3:09 pm
Another “poster child” might be a mid-ranked program that has improved dramatically in the last five years. @brite – this point is only good news to someone coming from a well-rated program, no?
teacherspaddle - October 6, 2010 at 5:09 pm
At our large state university, no one knows how the central office that supplies the stats even got their highly inaccurate stats. They didn’t ask for data from our department. None of the data is remotely correct. The campus dept that supplied the data never sent the data to departments for review or any kind of verification. I imagine the whole campus is shocked by the results, and some heads are rolling. But the bottom line is the data for our program is completely wrong.
arrive2__net - October 6, 2010 at 6:10 pm
Four years is just too long of a delay, especially in the ‘computer age’. All the programs are rated on four-year-old data, not just OSU. I think valid ratings would have to be conducted by an independent organization, and although the NRC (which is, as I understand it, a “quasi-governmental organization”) seems to be independent enough, taking so long to get the job done is not good enough. I have doubts about the validity of any of the rating/ranking systems, since I don’t think the grad schools occur from greatest to least. Instead I think they cluster around various levels of achievement, and that the level of excellence can vary even over the course of a single year.Bernard SchusterArrive2.net
mdslater - October 11, 2010 at 8:27 am
We are writing in response to David Glenn’s October 5, 2010 news post about our program as being representative of the problem of using NRC’s 2006 data to characterize the quality of today’s doctoral programs, given that about half of our faculty has been hired post-2006. Mr. Glenn acknowledges that it is possible our program is stronger than it was in 2006 given the ability to make new hires to an already-strong program, but suggests that our citing NRC statistics about the ranking of our program may be misleading given the extent of recent changes within our academic unit. We recognize the importance of Mr. Glenn’s point about using 2006 data to characterize a fast-growing program, and appreciate his post for prompting us to provide more timely empirical data concerning at least one relatively accessible key metric.
As detailed in the NRC study, we were ranked first in the Communication discipline in research productivity as of 2006, with an average of .9 ISI publications per faculty member per year across the years 2000-2006. We were at the time in the early stages of program building given a generous commitment by our College and University to help make us a leading program. Our 16 hires since 2006 included three internationally known and highly productive tenured scholars and the rest were outstanding and productive younger scholars. We have re-calculated that statistic using our current faculty as of fall 2010, looking at their ISI publications from 2003-2009, the most recent seven-year period for which complete data are available.
We have not been able to find the exact procedures used by NRC to count publications, but we understand that it involved checks of ISI data against the publications reported by faculty in the NRC questionnaires. We have sought to replicate the NRC procedures by using the CVs of our current faculty and a running list we keep of journals in which our faculty publish and whether these outlets are listed with ISI (with independent checks on journal ISI status). We included all tenure track faculty within our unit (N = 34) regardless of the nature of appointments to the graduate school. Despite this conservative approach, we find that the publications per faculty member per year in that most recent 7-year period, with the scholars who now comprise our faculty as of fall 2010, have climbed to 1.147, a 27% increase in average, annual ISI journal publication productivity compared to our discipline-leading 2006 NRC metric.
In other words, Mr. Glenn’s primary argument is in one sense supported: use of 2006 data in our case may be not an entirely accurate representation of where we stand in terms of faculty research productivity. Our analyses reveal our program has continued to move forward on this key metric since the NRC data were collected.
Obviously, the informativeness of ranking data would be improved by more frequent snapshots of readily analyzed archival data such as that on Web of Science, which may be the overarching point to take away from Mr. Glenn’s post (some of the earlier postings acknowledge this. In any case, we feel it is important, since we are used as the example in his article, to make sure that the discussion doesn’t leave any mistaken impressions about The Ohio State University’s School of Communication and the continuing progress we are making toward our goal of being one of the leading doctoral programs in the discipline.
Lance Holbert, Director Graduate Studies and Michael Slater, Social and Behavioral Science Distinguished Professor