The Chronicle: Colloquy Live Transcript

The Future of the SAT

Thursday, October 25, at 2 p.m. U.S. Eastern time

Is the SAT likely to survive in its current form and with its current level of influence in the college-admissions process?

The SAT is under more criticism today than ever in its history, with prominent universities diminishing its influence in the admissions process and in some cases considering ending its use completely. A number of efforts have started to come up with new admissions tests or new approaches to college admissions that don't rely on standardized tests. At the same time, many competitive colleges continue to require and praise the SAT, and to say they have yet to find a suitable alternative.

» The SAT's Greatest Test (10/26/2001)

John Katzman is the president and founder of the Princeton Review, from which more than 100,000 students a year receive preparation for standardized tests. He also created the Princeton Review Foundation, which helps disadvantaged students prepare for and pay for college. With Steven Hodas, he is the co-author of Class Action: How to Create Accountability, Innovation and Excellence in American Schools (Random House, 1995). Mr. Katzman will respond to comments and questions about the future of the SAT on Thursday, October 25, at 2 p.m. U.S. Eastern time. Advance questions are encouraged and may be posted now.

A transcript of the chat follows.

Ben Gose (Moderator):
Hello, and welcome to Colloquy Live, The Chronicle's real-time discussion forum. I'm Ben Gose, a reporter and editor here, and I'll be moderating the discussion.

Today, we're talking about the future of the SAT, which is under more criticism today than ever in its history, with prominent universities diminishing its influence in the admissions process and in some cases considering ending its use completely. A number of efforts have started to come up with new admissions tests or new approaches to college admission that don't rely on standardized tests. At the same time, many competitive colleges continue to require and praise the SAT, and to say they have yet to find a suitable alternative.

Our guest today is John Katzman, the president and founder of the Princeton Review, from which more than 100,000 students a year receive preparation for standardized tests. He also created the Princeton Review Foundation, which helps disadvantaged students prepare for and pay for college. With Steven Hodas, he is the co-author of Class Action: How to Create Accountability, Innovation and Excellence in American Schools (Random House, 1995).

We received lots of advance questions and I hope to get to several more in the next hour. Thanks for joining us, Mr. Katzman.

John Katzman:
Thanks so much for having me here in the ether today. I look forward to answering your questions. One thing I want to make sure is understood. We're not against tests; we're against bad tests. We work with a lot of tests at the sub-atomic level, and have consistently maintained that the SAT was a bad test. After all the debate about bias and predictive validity, what makes the SAT bad is that it has nothing to do with what kids learn in high school. As a result, it creates a sort of shadow curriculum that furthers the goals of neither educators nor students.

Question from Ben Gose:
The SAT has been controversial for decades, but seems to have weathered most of the criticism fairly well. Nonetheless, you're very pessimistic about the future of the test. Why?

John Katzman:
The SAT has been sold as snake oil; it measured intelligence, verified high school GPA, and predicted college grades. In fact, it's never done the first two at all, nor a particularly good job at the third. And it's done this mediocre job at a great cost of dollars and time on the part of students while blocking the development of all around better ways of picking a freshman class. Now that states are finally getting serious about measuring how good a job their schools are doing it's even harder to defend an admissions test that flat-out claims to test nothing kids have studied in high school. This momentum, reflected in the proposals by the UC and UNC systems to do away with the SAT in favor of state-based tests is probably going to sound the death-knell for the test. If that happens it'll be in part because the credibility of the test, and of ETS and the College Board in defending it, has been so weakened over the past twenty years or so.

Question from Ben Gose:
You're certainly not alone in believing that colleges would be better off using state assessment exams, rather than the SAT, in the college-admissions process. But one concern that's often raised is how a selective college that draws applicants from all 50 states would be able to make sense of scores on as many as 50 different state tests. Given the variations on each test by race, gender, etc., someone would have to develop an extremely complicated equivalency chart, right?

John Katzman:
An equivalency chart would be used, yes. A very complicated one, no. It's a straw-man to say that the stumbling block is the inability to compare 50 state tests with five-decimal point precision. What's needed, and will quickly be developed by a whole host of organizations, is "pretty good equivalency"--one that allows colleges some confidence that, more or less, an 172 from North Carolina is close enough to an 28 from California that they can base their decision on the rest of the applicant's portfolio. This is exactly the kind of thing the College Board ought to be helping its members with, if they weren't so culturally and financially wed to the SAT.

Question from Eric Loken, Pennsylvania State University:
By leading the call for more tests and more testing, isn't the Princeton Review just trying to manufacture new markets? Is the recent call for the abolishment of the SAT related to the erosion of Princeton Review's market share in the SAT test prep market and the encroachment of low-cost providers in that space?

John Katzman:
First off, the number of kids who want to go to a selective college is the same, however different the tests from one state to another. We're not calling for more testing by any means; since kids are taking the state tests anyway, we're trying to get rid of the extra ones. Whatever testing is done (whether within K-12 or for admissions purposes) should be done well; it should serve as well as possible the purposes for which it's intended, and the burdens should not be unreasonably shifted from institutions onto the backs of kids, as has certainly been the case with the SAT over the years. Certainly The Princeton Review has a business in helping kids (and increasingly schools) deal with high-stakes tests, but that business grows regardless of what test is in place. Our self-interest on this issue is far less than that of ETS or the College Board, for example. If you look at our words and deeds over the past twenty years, you'll find us a consistent voice in speaking up for intelligent and fair testing systems.

Question from Martha Phillips-Patrick, doctoral student and college counselor:
How can educational progress across all states be assured if that progress is measured by 50 different tests? What national standard will exist that measures progress in Mississippi against that in Iowa? If the SAT serves no other purpose, it does offer a national standard that makes State to State comparisons possible.

John Katzman:
NAEP was created to compare states. The SAT was never suited to that purpose, esp since so few kids take it in many states. For the majority of kids who go to college in their own state, the state test will suffice. For the others, colleges already have GPA, extracurriculars, etc. A pretty good equating into the familiar 200-800 point scale (which is very easy) will easily be enough.

Question from Richard Handelsman, Lithia Springs High School, Georgia:
If wealth determines SAT scores, why do African-Americans from middle-income families score lower than whites from low-income families?

John Katzman:
The issue of bias is tricky. It's not about who scores well or poorly--it's about how well SAT scores predict college grades. Women, for example, have higher grades in both high school and college, but men have far higher SAT scores. The SAT underpredicts their scores, as it underpredicts the scores of minorities and low-income kids. This is one of those problems that would not be hard to fix--if the College Board decided to fix it.

Question from Ben Gose:
Many of the efforts related to finding alternatives to the SAT seem motivated by the attacks on affirmative action -- court rulings barring affirmative action make colleges uncomfortable with an admissions measure that seems to limit the number of minority applicants. If black and white students scored similarly on the SAT, do you think this controversy would go away?

John Katzman:
Absolutely. Banning affirmative action turns the SAT into a set of handcuffs for admissions professionals. College admissions was never about rewarding good kids, any more than casting a movie is about rewarding good actors--it's about building a class that will learn from each other. However, state tests will give control of the process back to admissions officers in other ways. What sorts of skills and behaviors do you want to encourage? The test really does wag the dog...

Question from David M. Hallowell, Drexel University:
If colleges reduce their reliance on the SAT, what impact will that have on the GRE? Is its significance likely to dwindle, too?

John Katzman:
Colleges have actually been increasing their reliance on the SAT over the past 15 years, according to NACAC surveys. They need a common yardstick to make the process coherent and reduce grade inflation and manipulation. As a result, the only way to get rid of the SAT is replace it with some other yardstick that doesn't some of the SAT's weaknesses. State tests do this. Only when something comes along to replace the GRE will grad school lessen their use of it.

Question from Ben Gose:
Do you think the criticisms of the SAT I also apply to the SAT II, which is more focused on academic knowledge?

John Katzman:
No. The SAT II tests are curricula-bound, and therefore better. On the other hand, though, the SAT IIs are an embarrassing set of tests, especially when they're stood up against the APs. They're the cheap, cruddy version of real curricula-bound exams.

Ben Gose (Moderator):
We're now about halfway through the chat. Keep the questions coming for John Katzman.

Question from Barbara Hill, secondary teacher:
I have been looking over SAT I preparation materials and am quite concerned that some of the "answers" to the reading selections and vocabulary are questionable. With reference to vocabulary, I have observed that some of the "answers" reflect a somewhat superficial understanding of the words. Students should not be penalized for making higher distinctions, should they?

John Katzman:
The key to good SAT prep is that you're looking for the best answer and not the right one. It's about process of elimination, and keeping kids from overthinking on relatively staightforward questions. The SAT doesn't penalize higher order thinking, but it doesn't reward it, either. This is a test of elementary and middle-school skills, with a thick veneer of testmanship.

Question from Richard Handelsman; Lithia Springs High School; Georgia:
This discussion centers around college admission. But what about college graduation rates for those admitted through affirmative action or because they have "overcome adversity"? Do those students do as well as those admitted based on SAT scores and GPA?

John Katzman:
There is not great research here, and there are a lot of factors at play besides academic strength. What are the support systems at the school for minority kids, for example? Do they feel as comfortable and welcome as other freshmen? It's also an unfair question to ask of any one group. What are the graduation rates of legacy kids and athletes? To answer your question, those groups tend to do worse than kids admitted solely on the basis of academic performance.

Question from Eric Loken, Pennsylvania State University:
Princeton Review has historically presented itself as the "testbuster" and charged high fees to help students "crack the test". Now it sounds like Princeton Review wants to work with state governments to support their tests. Is your company now an education provider? Do you think the educational community will be receptive to that? Should the states be concerned that Princeton Review will be as combative against them as it has been to the College Board?

John Katzman:
That's somewhat unfair. If you look at our USMLE (medical licensing) or MCAT (medical school admissions) courses and books, you'll find no test-busting or bashing. We just review anatomy or organic chem. We teach what's on the test. And the difference between a 620 and a 760 on the SAT is almost always a facility with the idiom of the SAT itself.

Question from Ben Gose:
Do you think more states will adopt "X-Percent" plans for admissions? If they do, could that eliminate the need not only for the SAT but for alternative tests? What do you think of the "X-Percent" plans?

John Katzman:
I like the x-percent plans a lot, though we won't have a lot of outcomes data for a few more years. In many ways, it replaces affirmative action with something more acceptable to the public. These plans, though, do not mean we should keep the SAT or ACT. With all these new state tests springing up, we just don't need them anymore.

Question from Ben Gose:
Do you think colleges that stop requiring the SAT are hurt by their decision? While the SAT is still around, does it make sense for colleges to do this?

John Katzman:
So far, they're not. In fact, their US News rankings generally rise--their weak test-takers don't send in scores, and so they only calculate and report the average of their strong test-takers. In the end, SAT-optional is not compelling to me. It's another work-around to an easily solved problem.

Question from Eric Loken, Pennsylvania State University:
How is the GMAT a fundamentally different test from the SAT? You recently boasted to investors about gaining market share in the GMAT relative to Kaplan. You don't speak out much against the GMAT -- but to be consistent shouldn't you criticize it as well?

John Katzman:
Wow! Who are you? The GMAT is not a great exam, but it's taken in a different context than the SAT. These are 20-28 year-olds with significant self-confidence. They are on the way to completing a college degree, or they already have one. And many have a work-record that is tough to manipulate in the way high school grades can be. Could we find an exam that predicted B-school success better than the GMAT? Probably. Would it mean more black or female investment bankers? Perhaps. But it's just not a burning issue.

Question from Ben Gose:
Some people believe that the ACT is more closely tied to the curriculum than the SAT. Do you buy that, or do you agree with the College Board's claim that the two tests are very similar?

John Katzman:
It is somewhat more closely tied, but the College Board is right--the ACT quietly shifted away from content in the early 90's. ACT has a different philosophy, though, about this convergence between high school graduation tests and college entrance tests. While ETS sets up a for-profit division to compete in the graduation test market, ACT is quietly convincing states to make its current test into the graduation test. In both Illinois and Colorado, the ACT is a substantial part of the state graduation requirement.

Question from Jodi, Fairfield Univ:
Is there any research that looks at how well the SAT predicts GPA while controlling for course selection? Many would argue that the majors dominated by men have more conservative grading systems.

John Katzman:
Reading the research surrounding the SAT is a descent into hell. The College Board and ETS sponsor research that adjusts for everything, and finds the SAT a fair predictor. I dunno, but the ACT predicts college grades as well as or better than the SAT, but has a much smaller male/female gap.

Question from Ronald Iltis, University of California:
How is a system which ignores standardized tests in admissions in favor of race/ethnicity different from the numerus clausus?

John Katzman:
I think you mean quotas. And affirmative action really is different, if not in law (depending on the state), then in practice. Imagine you were looking at two kids, each with the same test scores. One graduated at the top of his class in the some inner-city school where 5% of the kids went to college. The other graduated in the middle of some suburban school. Who would you take? It's tough to be an admissions officer and have to defend something that's obviously the right thing for both the school and the kid. But the test scores don't take into account things that he/she is paid to look at.

Question from Ben Gose:
On an unrelated topic, how are your efforts to provide college-admissions counseling faring? There seemed to be a lot of hype two years ago about large for-profit companies getting into this business, but we haven't heard much lately.

John Katzman:
Our approach to college admissions has been--and remains--as follows: We do not compete with independent counselors, or counselors in well-staffed private or wealthy suburban school. We can't offer their experience and familiarity with the family. However, we can help disadvantaged schools and districts--whose ratios are well over 500 students to each counselor-- dramatically improve their counseling, through professional development, mentoring, and electronic tools. And in this area, we are making real headway.

Ben Gose (Moderator):
That's all the time we have today. Thanks to all those who sent in questions. And thank you, Mr. Katzman, for joining us today.

John Katzman:
Thanks so much for having me, Ben, and for everyone's excellent questions. Sorry I type so slowly! If you have any other questions, please feel free to email me at JohnK@Review.com.