From the issue dated June 6, 2003
|
|
Buyers and Sellers of Education ResearchBy J.E. STONEFor investors, buying shares in Enron turned out to be a bad choice.
The institute replaces the Department of Education's Office of Educational Research and Improvement and joins two other new agencies -- the Office of Innovation and Improvement and the What Works Clearinghouse -- in an effort to increase the scientific rigor of the research that educators and policy makers rely on. Improvement is definitely needed. But will the new agencies avoid the pitfalls that doomed the efforts of their predecessors? Policy makers need to understand why past research has served them so poorly. Organizations like the Office of Educational Research and Improvement, the regional education laboratories, the state education agencies, university research centers, and many other groups supported by public funds have been advising policy makers for decades, yet they have rarely detected programs that have proved to be badly flawed. On the contrary, they have rarely found fault with anything that someone was willing to pay for. Thus, over the years, policy makers have invested in countless innovations and reforms in education, but few of those investments have produced substantial improvement -- and many have been conspicuous failures. Virtually all, however, have been accompanied by favorable research, little of which was challenged by the various agencies and institutions that accept public funds to study and report on education research and policy. Nor have those agencies shown much interest in analyzing failures so as to avoid repeating them. Why? Conflicts of interest have been endemic in education research. The National Diffusion Network, a Department of Education program that was terminated in 1996, provides a classic example. The NDN's mission was to disseminate information about exemplary school programs to local school districts. In the late 1970s, it recommended several compensatory-education models based on the Follow Through Project, a set of rigorous, large-scale trials similar to what the new Institute of Education Sciences hopes to promote. However, instead of recommending only those models that proved effective, the NDN followed the recommendation of another federal body called the Joint Dissemination Review Panel and disseminated both the effective and the ineffective ones. The NDN was charged with disseminating the best programs, but educators who favored the ineffective programs -- because they were less structured and allowed more creativity on the part of the teacher -- set the JDRP's internal rules. Faced with the prospect of political backlash from the JDRP's constituency and from developers of the ineffective models, the NDN quietly sacrificed the public's interest in improved student achievement. The Institute of Education Sciences will face similar pressures. An instructive parallel can be drawn between conflicts of interests in research on education and those involved in Enron's demise. In the case of Enron, brokerage houses that provided the results of financial research to investors were also in the investment-banking business. They encouraged their analysts to give Enron a "buy" rating; in turn, Enron placed lucrative stock offerings for sale through the brokerage houses. The collapse of Enron taught investors a hard lesson: Sellers and buyers have competing interests. Sellers benefit from rosy assessments, but buyers benefit from candid ones. With little risk of discovery, brokerage houses will sacrifice investors' interests for their own. It is a lesson that education's investors -- that is, policy makers -- need to understand. The information that policy makers get is typically portrayed as impartial and objective, yet the parties that produce and disseminate it often benefit from neutral or positive reports. For example, the NDN's decision to punt minimized displeasure among its constituencies. If a source of research-based advice rarely finds fault with policies and innovations, it is unlikely to be serving the interests of education's investors and consumers. Self-evaluations are an obvious example. They may be useful and commendable, but they involve an inherent conflict of interest. A less obvious but equally potent form of conflict exists when programs hire outside contractors to assess their performance. For example, the National Board for Professional Teaching Standards recently awarded $6.6-million to 22 organizations to study various aspects of its teacher-certification process. Exhibiting some sensitivity, the NBPTS engaged the services of RAND to vet the proposals. Still, what is happening is that a program is seeking validation from researchers who will benefit if it prospers. If researchers find even modest support for the NBPTS's validity, the board will remain able to receive funds and will require the services of those researchers for years to come. The tendency of researchers who receive government funds to see no evil is reflected in the scarcity of their studies of questionable or failed programs. Most such studies come from private foundations and think tanks. Last year I carried out a study of 16 Tennessee teachers who are certified by the National Board for Professional Teaching Standards. Tennessee has an accountability system that objectively measures the ability of teachers in the third to eighth grades to improve students' achievement. Of the 41 teachers in the state who have been certified by the NBPTS, 16 have scores in that system. All 16 were rated as only average, not exceptional -- as their certification implies. The study was small, but it remains the only one to examine the relationship between NBPTS certification -- which often results in substantial raises for teachers -- and objectively measured student achievement. Moreover, it is the only one that suggests that the NBPTS should stop certifying teachers until it can clearly demonstrate the value of its credential. Given my findings, I expected the NBPTS to be displeased. What I did not expect was a hurried announcement from an admirer of the NBPTS, the Education Commission of the States, that it would convene a panel of "unbiased, distinguished educators and researchers" to review my study. That panel concluded that my results might not be widely applicable, yet the central finding remained: In 16 of 16 cases, NBPTS certification was no assurance of high-quality teaching. The implication for policy makers who had invested in the NBPTS was obvious. The conflict faced by the ECS was obvious as well. In a letter accompanying the panel report, ECS President Ted Sanders acknowledged that there was no research showing that NBPTS-certified teachers are more effective than others; yet he carefully avoided suggesting that the absence of evidence was any reason for policy makers not to spend millions of dollars on the NBPTS certification program. By taking a wait-and-see approach, organizations like the ECS appear to be prudent and cautious. In truth, they are making a virtue of necessity. They attempt to represent both buyers and sellers, and they have to straddle the line between them. It is inevitable that education initiatives with millions of dollars to spend will attract lots of friends. Resisting their influence will be a tough challenge for the Institute of Education Sciences and its sister agencies. What can increase the new agencies' chance for success is increased awareness by policy makers of what has caused past failures, and a more sophisticated approach to finding investor-friendly sources of advice on education. Like the now-chastened Enron investors, policy makers need to give greater weight to sources that have minimal conflicts of interest, and acknowledge those; that have a sound record of policy assessments; and that are demonstrably faithful to the interests of investors and consumers -- that is, sources that issue both "buy" and "sell" recommendations. Here are questions that a policy maker might ask to evaluate potential advisers:
Policy makers need watchdogs that bark, not agencies whose desire for self-preservation overrides their loyalty to investors and prevents all but the most oblique criticisms. To find out whether the new agencies in the Department of Education are loyal to education's investors, policy makers will need to pay attention to their performance, not their rhetoric. J.E. Stone is a professor of education at East Tennessee State University and the founder of the Education Consumers ClearingHouse. http://chronicle.com Section: The Chronicle Review Volume 49, Issue 39, Page B12
|
Copyright © 2003 by The Chronicle of Higher Education