The Chronicle of Higher Education: Colloquy

Partly because of prodding from the Bush administration, research in elementary and secondary education is increasingly organized around randomized experiments. Is that a misguided trend that ignores the realities of the classroom, or is it a long-overdue attempt to make education research truly rigorous? Read more ...

It is naive to expect that any one research methodology will do the job for studying elementary and secondary education or any other topic. It all depends on what you want to study. There may well be some aspects of education that can be quantified, controlled, randomized; but there are equally other aspects that would be distorted by being forced into such a mold.

It would be ironic just at a time when qualitative studies are proliferating in the health care fields that education should abandon the methods that have yielded important insights in the past in favor of a lock-step quest for "scientific rigor." Randomized clinical trials are great for studying the effectiveness of drugs, but they are not well suited for learning about many important aspects of the healing process.

The argument that one exceptional teacher could skew the results of a randomized study to the extent that the results would have no "internal" validity suggests that the effect size of the independent variable (the treatment) is trivial. Nonetheless, the unit of analysis should be the classroom and not individual children. That is, a study conducted with a 1000 students in 50 different classrooms (e.g., with 20 students in each class) divided into a control and experimental group should be evaluated using the mean value of the dependent variable for each classroom. So if the sample sizes are equal, you would have n = 25 for the experimental and control groups. The error variance is between classrooms (within each group) and not between students and the degrees of freedom (for an independent t-test) would be 48 and not 998.

This topic is covered in Stanley and Campbell's classic text. However, most designs in education use the student as the experimental unit. Note, that if the above procedure is followed then the variation between teachers within a group will contribute to both error and treatment variance. One could of course, by chance, end up with all of the excellent teachers in the experimental (or the control) group but randomization makes this unlikely and that is the entire purpose of using experimental designs and statistical inferences.

Experimental design and statistics will not always lead to the correct treatment for everyone. Even in pharmacology, the drug or chemotherapy that works bests for the group may not be effective for every individual. Experimental design and statistics are about making better decisions. Unfortunately, social and educational policy decision are frequently (usually) made on the basis of theoretical and ideological beliefs involving the continual regurgitation of old ideas into new patterns reflecting the current conventional wisdom. This is compounded by the real difficulty of doing experimental research in applied settings.

Hey guys, let's get serious. Is this the most interesting colloquy subject you could find? This is the Chronicle of HIGHER Education and the principal connection between this subject and HIGHER education is that it concerns "education research." If we're going to talk about elementary education let's talk about something we can really sink our teeth into, like the fact that there's no correlation between what Americans spend on it and the results that are achieved. Let's talk about how the beloved seniority system of the teachers' unions penalizes teachers who relocate, as if we're still living in the pre-women's rights days when women could be teachers or nurses (teachers, that is, if they did not get pregnant or, in some locales, not get married). Let's talk about the high level of support for voucher programs in the African-American community and the manner in which politicians deny them such alternatives. We don't have to just look at so-called 'conservative' issues. Let's talk about the current opposition to teaching evolution. Where did the elected officials leading the charge come from? Who educated them?

I have no trouble with seeking and funding randomized trials. Indeed, I welcome increased federal funding of educational research of all sorts. There has been very little funding for education research considering this is such a high priority for our nation.

What concerns me most, however, is the phrase "scientifically based research". Taken superficially, I suppose this to mean that educational research should be conducted in a way that is like that of scientists. Science, of course, does not always use randomized trials. I'm not quite sure how a geologist would do that when faced with strata in a rock. She must work with what she finds and gather observational data as well as make other quantitative measurments. Descriptive research is a key part of doing science along with a wide variety of quantitative methods. Much of Biology over the past couple of centuries has involved descriptive research.

Actually, scientists use whatever methods will provide them with answers to their questions. They will use the evidence gathered (descriptive and quantitative) to convince colleagues that their results are correct and meaningful. The community of practice then determines the value and utility of the work.

It is safe to say that not all scientific research and not all education research has been meaningful. It may be that educational researchers are not conducting some of the types of studies that would provide the desired information. Long-term studies, for example, are a staple of medical research but are rare in education because of funding constraints.

But let me remind everyone that there is no "magic bullet". No single research approach that will answer all questions. The process of scientific inquiry is complex and iterative. I recommend the policy folks look more carefully at how science is done. I suggest also that my education colleagues consider the paper by Burkhardt & Shoenfield (Dec. 2003, Educational Researcher) regarding how educational research groups could be organized more like science research groups. Their suggestions would not change the type of project one might pursue, but the organization they suggest may allow certain questions to be more thoroughly and effectively addressed.

Having switched to Educational Leadership And Foundations at my alma mater, Indiana State University after completing my M.S. in Physics, I am now continuing in ELAF after obtaining my teaching degree in 1991. Talk about your social scientists...

The current trends in testing and accountability in the State of Indiana are yielding some unexpected results when compared with surveys revealing school culture and climate. As you may well know, school climate and school culture are not the same thing, but they are intimately related in any social structure.

Preliminary data suggests that those schools with the most positive climates and the most socially progressive cultures are outscoring their contemporaries on performance accountability testing like the ISTEP in Indiana. Undoubtedly, other testing will show the same results using other valid instruments and research scenerios.

This is not surprising to us who know that the environment of a school is a very significant thing and how you deal with people on a daily basis GREATLY affects ALL academic performance outcomes across the board. This is true in not only education, but also in business and industry. Similar studies in areas other than education will yield the same basic results.

However, the "Nation At Risk" supporting crowd has bashed the heads of those peole who supported these ideas in the legislative arena for at least two decades, now.

The "fundamentalists" and the "back to basics" crowd had better be prepared to ingest a generous serving of crow after this research is done, because from what I have seen, it is very consistant and repeatable.

These well-intended "back to basics" folk may have indeed fashioned the noose with which they shall be hung, with their accountability demands regarding performance testing.

We could draw direct parallels to what the Japanese automobile industry has done to the American automobile industry in the last 30 - 40 years and the difference in how Japanese companies as a rule handle employees. The list goes on and on. What is the trure reason that industry has "outsourced" jobs from America? Could they know this already?

In other words, we are in for some dramatic culture shock and drastic changes if we do what we need to do as a nation to maintain our post as leader of the free world.

...And you heard it here first...

Bill Harwood wrote:

"Actually, scientists use whatever methods will provide them with answers to their questions. They will use the evidence gathered (descriptive and quantitative) to convince colleagues that their results are correct and meaningful. The community of practice then determines the value and utility of the work."

Science is science because it is based on publicly observable and replicable experimentation. Harwood is certainly correct that not all science is based on randomized designs. Much of science is observational but when research involves determining the relative effectiveness of different forms of treatment (including educational interventions) then it becomes desirable to conduct experimental or at least quasi-experimental studies.

Geologists and paleontologists usually cannot use experimental manipulation because they are investigating historical events. A better comparison for education and psychology would be field research in biology. For some events the researcher can only observe and record but experimentation is applicable in other situations. Medicine, agriculture, and biology are examples of disciplines dominated by experimental research and the progress has been impressive. Unfortunately for the social sciences and education, I doubt that we have control of variables that have effect sizes comparable to those of antibiotics or crop hybridization nor do we have theories with the explanatory power of evolution.

I am astounded that anyone in the educational research community would object to the idea of conducting more "scientifically-based research" to improve educational practice. As the recent National Research Council study entitled "Scientific Research in Education" pointed out, "Scientific research in education can shed light on the increasingly complex and performance-driven U.S. education system." Moreover, the NRC study indicated that "At its core, scientific inquiry is the same in all fields. Scientific research, whether in education, physics, anthropology, molecular biology, or economics, is a continual process of rigorous reasoning supported by the dynamic interplay among methods, theories, and findings."

Surely, randomized experiments have a place in education inquiry. But as in every other form of scientific study, the appropriateness of the methods used depends on the nature of the research questions under investigation. In the case of questions dealing with causation, experimental and quasi-experimental research designs may indeed be most appropriate. But as Campbell and Stanley reminded us in their classic "Experimental and Quasi-Experimental Designs for Research," these types of studies are subject to substantial threats to internal and external validity and whatever scientific claims are made based on the results can be strengthened significantly through testing by multiple methods.

Fundamentally, the problem with the quality of educational research is not a lack of appropriate methods but rather a lack of adequate funding. And I hope that the current attention to educational research will lead to a greater investment of federal research dollars in education. I am also mindful, however, of the disconnect between a federal policy that insists on funding only program initiatives supported by scientific research and their funding of policy initiatives totally devoid of scientific evidence. This only hurts the credibility of what is an otherwise noble and much needed call for more and better educational research.

The faculty of the Indiana University School of Education has engaged in its own internal conversation about the proper role of scientific research in education. I believe this is a healthy debate that can only improve our effectiveness as educational researchers and scholars. And I only wish that our nation's leaders would likewise engage in an open, unbiased dialog about the proper role of government in advancing educational research and improving educational practice.

If education were univariate in outcome the use of randomized trials might be more useful. As has been noted by many scholars, we expect our schools to accomplish many different outcomes. Although there is agreement that young people should be developing basic skills, there is also strong support for the development of more abstract thinking skills, development of vocational skills and attitutes, development of citizenship and development of personal fulfillment. The problem with a focus on a single outcome variable is that we lose the effects on the other desired (by many according to Goodlads work from the 80s) outcomes.

Even if we could -- identify/measure/control/facator -- all the covariates of the treatments, we would have great difficulty in even measuring the multiple desired outcomes.

If we are going to look to the sciences for our models we ought to be looking at ecology (not medicine)--- a field where the models are messy (often stochastic) and the finding very contextual or local. It is a field where grand theories have fallen by the wayside and local science has been melded with local activism to achieve desired outcomes. It provides interesting analogs to education.

I can't imagine where the Chronicle got the idea that randomized studies have increased in education--the available data certainly don't support that claim, despite the proddings of Russ Whitehurst and the randomized trials idolators at the US Dept. of Education. Daniel Robinson, of the University of Texas-Austin, presented data at the American Educational Research Association meeting in San Diego in April of this year, showing that publication of experimental studies in five of the major educational psychology journals (most likely to publish experimental education research) and the American Educational Research Journal has remained very stable over the past decade. About 15 percent of articles published in AERA are experimental; about 25 percent of articles published in five ed psych journals (e.g., Journal of Educational Psychology, Contemporary Educ Psych) are experimental. There has been no dramatic increase (in fact, no real increase at all) in published experiments. Given the realities of schools, the costs of doing experiments, and the lack of available funding to support such work, we will not see any increase in these studies any time soon.

Don't be so passive about the utility of "randomized trials." Now that medicine is doubting their absolute value, one could well ask education to justify them too, beyond the argument of a statisitcs 101 course.

Experimental research in educational interventions do not seem to render results that are significant in a consistent and generalizable manner. Just because a difference may have been noted between groups in a given study does not mean that there's always a high enough probability that the difference was due solely to the intervention. (Usually it's not.)

Another problem seems to stem from a lot of incomplete reading and analysis of research studies. Reading the full concluding remarks of research reporting is REALLY important---but a lot of reporting for large "lay" audiences offer what is really a distillation of the abstract. Important caveats and insights into other questions which arose from the experiment are often omitted in the more public discussions. It's as if somehow the public isn't trusted enough to be able to follow what researchers mean by a good explanation of the words "not statistically significant."

The problem with using randomized experimental studies to guide any sort of policy is that it's a misuse (and often a grave misinterpretion) of what you do with "results". Inferential statistical analysis can allow us to make educated and reasonable conjectures---but we shouldn't be drawing hard and fast conclusions about anything based on experimentation until after a long time of replication has elapsed that yields very similar "results" with a high degree of agreement among the stakeholders. However, when this is pointed out to policymakers (who understandably have to provide short-term answers to constituencies who are paying a high premium for educational services) you run into the problem of having to justify everything from metholodogies to curricular choices to testing and standardization and so on.

So we wide up with a well-meaning policymaker reading in a media source that "such-and-such intervention" raised test scores and the important part gets left out---that very little of the variance is due to the intervention itself, making it very hard to justify using it in "non-lab" settings. This usually means the funding for that particular research project drys up, and the research team moves on to something else that will yield hopefully better fruit. The former intervention certainly shouldn't be used to engineer some sort of nationally mandated program or set of guidelines for instruction or testing or what have you---but it shouldn't necessarily be wholly discarded either.

Re-design of research trials is something that I think needs to be put on the table if the national debate about this is going to be remotely meaningful. How many times has a basically good research question foundered due to a lack of follow-up experimentation? The questions are thus left hanging and sometimes really good ideas and insights dry up. And it's not just due to lack of funding, though that's a big part of the issue. Severe communication challenges and limitations between educators, administrators and researchers; pressures of continuous publication; the necessity of producing "useful" data, i.e., results that can be used to not guide but produce policy (which I feel is a very dangerous undertaking); and the availablility of populations sufficiently large enough to take into account generalizability are just some of the other issues that are really problemmatic with regard to using experimental research results to guide policymaking in education.

"If we are going to look to the sciences for our models we ought to be looking at ecology (not medicine)--- a field where the models are messy (often stochastic) and the finding very contextual or local. It is a field where grand theories have fallen by the wayside and local science has been melded with local activism to achieve desired outcomes. It provides interesting analogs to education." Paul Rowland, Paul Rowland, U Montana

I really like this analogy: I think the idea of looking at ecological studies is a very cogent way to look at this issue---schools ARE ecological in that they are social, local, and political communities where many stakeholders' goals crisscross and often clash. This is regardless of whether they are places of primary, secondary or higher education. In this sense, the interplay of local, national and international pressures and goals always have to be borne in mind as we not only set policy, but as we design research projects as well.

Tamara,

You model is well thought out.

This is especially so in light of the circumstance of the conservatives and the politicians giving the public a view of education that could be compared to the exaggeration of the ecological implications of the recent film "The Day After Tomorrow".

Public education and education in general in America is not in any more trouble than is America society. If you want to impact education you must first change the attitudes, cultures and climates of Americans in our cities, towns and neighborhoods.

The problems we see in public education do nothing more than indite American society in general. The public is very susceptable to manipulation on both sides of the public education issue, but public education is a creation of our society.

We would need a whole new discussion to outline these problems. Part of the problem is with the fact that education, inquiry and learning take a back seat to such things as sports and public entertainment. In fact, there is an undertow of a mob attitude toward education. Public education is well-entrenched in our society. It must be changed from within by intellectual persuasion. It will never be changed from the outside by brute force.

Indeed, until everyone pulls together in the same direction public education will not change. Until we learn that not everything in America must be torn down to the ground before it can be re-built, the morass in public education will continue with little progress being made.

This is true of Primary, Secondary and Post-Secondary education.

Lab coats in grade schools