The Madness of Rankings

The time for the annual ritual of releasing rankings of world higher-education institutions has arrived. It is a season of celebration for some, indifference for a few, and, of course, upset and criticism for a substantial number whose institutions are not listed on the famous roll of the notable and venerable. All entries in these selective clubs depend on which side of the do’s and don’ts table each institution finds itself as regimented by the tyrants of ranking.  

In case it hasn’t already become obvious, I am among those who view rankings with some cynicism due to their past misuses and abuses. At the same time, I must concede that they can be a useful tool to help guide institutional improvement.

Last week, I was in Bogotá where I spoke at a conference organized by the Colombian Network of Internationalization of Higher Education (RCI) and visited CONAHEC member institutions. My visit to Colombia coincided with the release of the QS World University Rankings, so I heard repeatedly that, with the exceptions of the National University of Mexico (UNAM), ranked 222, and the University of Sao Paulo (USP), ranked 253, Latin American institutions were notoriously absent among the ranks of the top 300 higher-education institutions in the world. Having come to this realization, I feared that Latin America would be literally wiped-off the face of Earth during the subsequent release of the rankings published by Times Higher Education, and I was almost correct: the sole survivor which figured in those was UNAM, ranked at 150. At least the honor of the region was saved by UNAM. Our institutions aren’t completely defunct … at least for one more year.   

Indeed, rankings are part of the nature of education. Like it or not, comparisons are unavoidable. Even in my small rural Mexican hometown of Ojuelos I still remember from when I was a kid the odious comparison made locally between the students of the Colegio “Miguel Hidalgo” (the private elementary school in which I had the privilege of being enrolled) and the Escuela del Gobierno (the public school considered to be of lower quality). If this happened in Ojuelos, why should I not expect something even more sophisticated on the world stage in higher education?

But although rankings may be here to stay, we must be aware of their many limitations, their intended and unintended biases, and their use by institutions and even national governments. For instance, “reputation” continues to be a major component of both the QS and Times Higher Education rankings. However, the formation of the group considered worthy of deciding this “reputation” is noteworthy in that it underrepresents many important parts of the world academic community. For instance, the Times Higher Education rankings gave a considerable (34.5 percent) weight to institutional prestige as defined through a “worldwide” Academic Reputation Survey which included 13,388 responses provided by faculty members presumably from across all world regions and subject areas. As can be expected, the basis for this survey included only 3.6 percent respondents from Latin America and 2.3 percent from Africa. Efforts have been made by Times Higher Education to reduce this bias by making the survey available to respondents in various languages, but much work remains to be done in securing responses from a sample with reflects the world’s diversity.

To further complicate the utility of these rankings, it appears that not a single perspective was included from outside the Ivory Towers. I suspect however that graduates of institutions working outside of academe may have something to contribute (ultimately one hopes they learned something at their universities), as would employers (who “suffer” through any limitations which accompany higher education graduates they retain).  Even parents (who pick up the tab for university) are likely to have something to bring to the discussion. What about the perspectives of students? Oh yeah, I forgot that students can’t provide a legitimate opinion about their institutions. What do they know about them? I guess not much since their points of view are flatly ignored. In the end, it seems their ideas at times become an unheeded casualty in higher-education institutions.

I once heard the semi-facetious characterization that the ideal university for research is one having everything but students – with the exception of those working as knowledge-slaves for researchers. Perhaps this is why rankings are not among the key factors my son and I are considering in our search for a university where he can pursue his undergraduate studies.

Another important component of rankings relates to their research impact. For instance, Times Higher Education rates this component by measuring volume, income and reputation of research (30 percent), research influence in terms of citations (32.5 percent) and even industry income (2.5 percent). Of course, it may be easily argued, especially by people outside of academia, that research relevance cannot simply be correlated merely with volume. Also, formulas like the ones used by The Times tend to perpetuate the “publish or perish” principle so sacred and untouchable in the world of higher education. As a side note, this year Alexandria University of Egypt made it into the Times Higher Education rankings for the first time, placing a decent 146th. This unexpected jump was mostly thanks to the weight given its citation index which, according to the report, was even better than that of Harvard University.

An interesting variation in the rankings’ madness is that introduced by the even more famous Academic Ranking of World Universities (ARWO) popularly referred to as the “Shanghai Ranking” and originally produced by the Center for World Class Universities at Jiao Tong University. Thanks to this ranking, additional value has been placed on Nobel-Prize recipients. These folks have suddenly become an even hotter commodity for higher education institutions. The proxy used by ARWO as a criterion to partly measure quality of higher education is a simple count of the number of alumni of an institution who have won a Nobel Prize or Fields Medal. In fact, 10 percent of the total Shanghai ranking is explained by this indicator. As expected, the proxy used for quality of faculty members includes whether or not staff of an institution have won Nobel Prizes and/or Fields Medals. Faculty quality as measured this way garnishes a significant 20 percent of the rankings’ total value. Thanks to these two indicators, the 829 Nobel Laureates the world has produced now have extra value, even those 515 who are already dead. Those recognized in this way can be counted at our convenience, even twice or three times if necessary.

A final interesting variation of the rankings is that provided by Webometrics, an initiative established in 2004 by the Cybermetrics Lab of the Spanish Scientific Research Council. Webometrics measures the Web presence of a university as a proxy for the quality of scholars and research institutions worldwide.  

Unfortunately, inclusion in the many “Who’s Who in…” publications, which inundate our e-mail systems with invitations, is not yet considered by any decent ranking. Perhaps some day a new ranking might include this very important and rigorous indicator of success.

In summary, the rankings game is puzzling and complicated. Are any of them worth considering? Obviously, it depends why and how these rankings are to be used. The central question all rankings seek to answer is:  Which institution is best? The problem lies in defining what is meant by quality and, more importantly, how it can be measured. Let’s consider the case of my own institution, the University of Arizona. Worldwide it ranks 95 in Times Higher Education, 160 in the QS Ranking, 45 in the ARWO, and 30 in Webometrics. Such wide variation plainly illustrates the importance of examining the methodology behind the rankings in order to make appropriate use of them.

My suggestion is to follow some of the recommendations that Jamil Salmi and Alenoush Saroyan made in a very interesting article published by the OECD Programme for Institutional Management of Higher Education Program (IMHE) entitled  “League Tables as Policy Instruments: Uses and Misuses”:
+ Be clear about what the ranking actually measures
+ Use a range of indicators and multiple measures rather than a single, weighted ranking.
+ Compare similar programs or institutions.
+ At the institutional level, use rankings for strategic planning and quality improvement purposes.
+At the government level, use rankings to stimulate a culture of quality
+ Use rankings as one of the instruments available to inform students, families and employers and to fuel public debates.

And last but not least I’ll add one – have fun with the rankings!

In any case, I will continue to enjoy the reactions these increasingly famous rankings provoke. An interesting one I came across last year was the declaration made by a legislator in Mexico who, when asked by local reporters why UNAM was excluded from the top 100 institutions published by Times Higher Education, responded very seriously that it was due to the secret conspiracy of powerful local political groups interested in discrediting the institution as a way to reduce its public subsidy. Lions and tigers and bears! Oh my!