"But the gains sure don't look like much. Second, students who start behind tend to stay behind;"

This is god's doing, or nature, or natural variation, yes?

"put another way those inequalities at the starting gate are consistent. Students whose parents have less education, and those from racial/ethnic minority groups have substantially lower test scores."

There is a book by a couple of Harvard guys called The Bellcurve; you should have a look.

t_paine, you're showing your ignorance of the topic. The better prepared are better able to take advantage of the same environment--"the rich get richer," effect has been repeatedly documented. As for "The Bell Curve," that was refuted a long time ago. I'd suggest you take a look at Neisser, et al.'s comments on it. (And it was written by one guy from Harvard; Charles Miller is a propagandist working for the American Enterprise Institute, which should tell you something else about The Bell Curve.

--Robert Dushay

Speaking of ignorance, it's Charles Murray. And knowing where he works saves us the trouble of cracking the book?

If we think that what the CLA measures is why students go to college, major in accounting, history, or mechanical engineering, we might as well go home now. When's the last time you heard them talk about learning how to make and break arguments at the Thanksgiving dinner table?

And if we think what the CLA measures provides a full account of higher education and documents student worthiness of receiving a degree, then let's set passing scores in every 4-year college (easy to do, though the passing marks will be different from school to school), require a passing score from ALL potential graduates as a condition of receiving a degree (and not just the 100 who volunteer to take the test), and give them 3 shots at the test. That would scare the wits out of everyone, no?
Enough said.

--Cliff Adelman

t_paine's comments are certainly ridiculous, though I won't continue that line of comment any more.

What I also find ridiculous is the extent to which people constantly get sucked into the "test score." Why do so many who are actually in it continue to push higher ed toward SOLs? I now have classes full of students who are great at achieving good looking test scores. But they can't think their way out of a paper bag. Nor do they know what it is like to do real mental labor toward real learning - but hey, they have good test scores. Beware.

All of us realize that any particular test score represents only part of the overall measure of learning. The Arum-Roska study examines an important question that all university faculty and administrators should be asking ourselves continuously: what skills, knowledge, and dispositions do our students gain as a direct result of their experiences at our schools? The study is a starting point. CLA results show only some variables, true, but we should be interested in those results as well as others. Would other indicators such as the GRE lead to the same conclusions? Is the growth (or lack of it) within indiviudal schools in the test group statistically significant?

I'm having trouble getting past the title of this article. Should it be, "Are our students learning?" Or is there something that I've missed?

How can we expect students to learn when reporters write ungrammatical titles like this? "Is" and "students"! Even grammar school students can match subjects and verbs better than this.

With the caveat that a lack of benchmarking does add a challenge to interpreting these test scores, nevertheless, the small size of the typical score changes seem disturbing. I base this conclusion on four assumptions:
1. The survey instrument is valid: A sample question presented on p. 22 of the report .pdf, available at http://www.ssrc.org/workspace/uploads/docs/CLA_Report.pdf ) appears to track the subject matter we would hope is being taught in the schools rather closely.
2. The average score increases eyeballed by Sara ranged from 35 to 50 points (3.1% to 4.1% increases of above the CLA score of 1125, respectively) during the 22-month longitudinal interval Sept 05-June 07)
3. Young people would be learning outside of college: Although there was no mention in the report .pdf of a negative education control group (identical longitudinal tests, administered to non-college students at the same ages as Freshman and Seniors assessed,) young people between the ages of 17 and 19 would be learning rapidly even without college due to the demands of jobs of increasing complexity, increasing personal responsibility and recent/impending financial independence.
4. Lacking baseline and control data, one may note that the average CLA score of 1125 per student at age 17 increased from 0 at birth an average of 66 points per year. It most certainly did not do so at 66 points each year; however, if the (missing) control increase during the 22 months between students aged 17 and 19 was ANYWHERE near this mean, negative control scores would have increased 121 points, or 10.8%. Compared to the study averages of 3.1 to 4.1% for the same period

While I do not claim that colleges are slowing the learning of our young people (though I do know individuals who energetically propose this) I do think that the larger question raised by this study is not how to partition the variability in the CLA longitudinal study, but whether our colleges are cost effective in comparison with on-the-job training. A control treatment would be very helpful in laying this argument to rest, or in raising provocative questions.

come on, people -- the title "IS" a joke! (presumably reflecting grammar skills of some students . . . )

Bah ... "is the CLA capturing the kind of learning we'd expect colleges to be teaching?" The value of higher education is development of the mind, something not so easily measured. We've now trained several generations of students to be adept test takers while stunting their ability to actually think through and solve problems. There is a big difference between identifying the "right" answer and developing an answer.

It's an allusion: former President Bush, in Florence, SC, January 11, 2000:
"Rarely is the question asked, is our children learning?"

Yes, and it does reflect the grammar skill of that particular C student.

Any idea what the test was testing, by way of "student learning"?


Will Fitzhugh
The Concord Review
www.tcr.org
fitzhugh@tcr.org

To cwinton and willfitzhugh, please read the short introduction to the CLA (College Learning Assessment) provided in the link in the first paragraph of the article. Students who complete that "test" perform one of two tasks: 1) they argue for a particular claim and then rebut an argument presented in the prompt OR 2) they use provided primary documents to propose a particular solution to a problem (called a performance task in the CLA's vocabulary). Neither kind of task has a canned response, either "correct" or "incorrect."

csmomaha - I think it was a deliberate reference to a misstatement made by a recent former President of the US.

I have always found the debate about standardized/external measurements of academic achievement fascinating, especially when someone in the discussion knows enough to make meaningful comparisons to other countries (I don't, but maybe someone familiar with the Indian system can say something here?).

If we think learning can't be measured on a standardized/objective test, why do so many university classes pin so much of their grades on exactly that kind of test, called a Final Exam (many of which are fill-in-the-bubble format)? If those are legitimate, then the issue isn't the assessment format, but rather the question of who decides the curriculum to be tested. If those aren't legitimate, we in academe have a heck of an internal problem we should be devoting a huge amount of attention to addressing.

With students going into what could be 6-digit debt just to get a bachelor's degree, maybe it's reasonable for them to insist that there be national curriculum standards for what that degree actually means in terms of the content of what they should have been taught - that in turn seems to me to be the obvious precursor to national external exams at the college level. We already see this in professions that have licensing requirements, like nursing.

If I were spending that much money today, I would want to know before choosing a college where its graduates rank in academic achievement compared with other colleges in my price/selectivity range. The graduate exams are only a poor proxy for that measure because only an elite few take those so it only affords comparison among the top students across institutions, which isn't relevant for most students needing to choose where to plunk down $100-200K and the next 5 years of their lives.

earlixd has outlined in a most compelling fashion what needs to be done. Almost all young people learn. Moreover, noncollege youth undoubtedly acquire some of the skills and knowledge that are being meaured by the CLA. If 30% of gains in CLA scores can be attributed to "college-level factors,"I would be interested in seeing the kind of gains delivered by the School of Hard Knocks which is not only open admissions but also tuition free.

If we don't find this question interesting, how are we different from Cesare Cremonini who famously explained his reluctance to peer into Galileo's telescope thus,

"I do not wish to approve of claims about which I do not have any knowledge, and about things which I have not seen .. And then to observe through those glasses gives me a headache. Enough! I do not want to hear anything more about this."

Interesting

I contacted Roger Benjamin, one of the study authors with my originally posted questions, and he forwarded my questions to the team. I have these additions based on their responses:

Jeffrey Steedle noted that I was "interpreting CLA scores as though they are on a ratio scale (i.e., 0 is the lowest score, and it indicates the absence of critical thinking and writing skills). The difference of 35 to 50 CLA scale score points actually reflects an effect size of 0.25 to 0.36 standard deviations (where the average within-school standard deviation is 139)."

I also asked about the possibility of comparing these groups to students who had not attended school (Joseph Foley's "School of Hard Knocks" U, comment above) and Jeff said they were considering Propensity Score Matching to do exactly that.

As Sara noted in the original article, these studies will be good to "keep an eye on."