Surveying the Rankings

This week admissions officers and high-school counselors throughout the nation received a survey about “America’s Best Colleges,” U.S. News & World Report’s annual rankings of colleges. The electronic survey was sent by an ad hoc committee whose members represent the National Association for College Admission Counseling and U.S. News. In other words, one very large membership organization and one very controversial player in college admissions.

The committee’s chairman, Peter E. Caruso, explains that the committee isn’t a partnership—just a joint attempt to spark a continuing dialogue about the magazine’s influential rankings. Last year the committee met at the association’s annual conference, in Baltimore. There was no shortage of critiques, to say the least. For instance, NACAC’s members raised concerns about the rankings methodology, the integrity of the peer-assessment survey, and even the guide’s name. Objecting to the phrase “Best Colleges,” the group’s members proposed “Great Colleges” as an alternative (don’t hold your breath on that one).

Mr. Caruso, associate director of admissions at Boston College, knows that some members of the association aren’t keen on the committee. “There are those who say that we shouldn’t engage with U.S. News at all, and I’m not sure this committee with change those opinions,” Mr. Caruso says. “Then there are those who have resigned themselves to the fact the rankings aren’t going away, and so it’s important to engage U.S. News and have a constructive conversastion.”

The survey was designed to further that conversation. The first section asks respondents to describe their opinion (agree, somewhat agree, somewhat disagree, or disagree) of various statements, including “U.S. News rankings are helpful resources for students and families interested in college information,” and “U.S. News rankings are useful to college and university recruiting efforts.” Another question asks: “If your institution is presented favorably in the U.S. News rankings … do you promote that ranking in your marketing strategies and materials?”

Mr. Caruso believes the survey will give the committee tangible data that could, perhaps, prompt changes in the rankings. The survey was sent to approximately 10,000 people, and so far about 1,200 have responded.

But who might the survey help? Mr. Caruso has considered the question. “Who benefits from this?” he says. “The families? The students? The profession? Or, ultimately, does it benefit U.S. News? That’s something we as a committee have wrestled with since Day One. Having said that, I don’t think there’s a reason not to engage them.”

U.S. News did not play a role in developing the survey, and only one question deals explicitly with the rankings formula. Nonetheless, Robert J. Morse, director of data research for U.S. News, says he will be interested in the survey’s findings.

“This would be a sophisticated look at the rankings from a different perspective, something that goes beyond the anecdotal about how they really feel” he says. “There are certainly outspoken people who are opposed to rankings, but there hasn’t been something that gives a comprehensive view of this constituency.”

Mr. Morse could only guess what conclusions U.S. News might draw from the survey. “Maybe it will turn out that even though everybody complains about the rankings, everybody uses them,” he says.