Video Lectures May Slightly Hurt Student Performance

I have to emphatically second what nordicexpat said – watching a lecture online is not online learning. This is a sad comparison.Did they control for how boring the videos were, anyways? If this is the highest-quality evidence…. then reserach on this subject is in sad shape.

This research did NOT compare online courses vs face to face courses! “Students …were randomly assigned to live lectures versus watching these same lectures in an internet setting, where all other factors (e.g., instruction, supplemental materials) were the same.” It compared live lectures vs recorded lectures. That’s ALL it compared–”The Sage on the Stage” live, or “The Sage on the Stage” recorded. If the Chronicle felt this research was really newsworthy, then I would expect a much better analysis than the opening line of this article.

If this is the “highest-quality evidence that’s available” then we have nothing to go on. Frankly, the reporting on this kind of study needs to be a little more insightful, or we need a better analysis of what was actually done. On-line video is not on-line learning.

Slight effects, marginal correlation rates, without boundary conditions and magnitudes in practice situations where all things are never equal (self selection abounds, status denigration abounds, JSBrown’s serendipities abound, etc.) mean nothing in usual social science and mean nothing in these eLearning versus Colleges studies. Yet it is good to get “news” on how each badly designed and terribly reported study is “done”. Nice to know professors are not competent at research, reporting, and eTeaching.

Yup, I hate to be a “me too,” but I think those involved in the study and the Chronicle need to be aware of just how ludicrous the “study” actually is. And published by the “National Bureau of Economic Research” – ah, yes, a major player in the field of education. The NBER is dedicated to, according to their website, “promoting an understanding of how the economy works.” I would have thought that they have their hands full right now, and have no need to dabble in other areas. Worst is, it will probably be cited in many journals as a truly terrible piece of work, but, because it will have so many citations, will show up positively on the stats. Best is, you can purchase the original paper for only $5.00! :-)

I would add that the Department of Education study mentioned in the article was another disturbingly flawed example of pseudo-science. Not only was the study itself a poorly-designed “meta-analysis”–hence, not primary controlled research–but it was reported by the media as being somehow relevant to evaluating the effectiveness of on-line education in colleges and universities in general, even though nearly all the studies it was based on were of small-scale courses that were not part of undergraduate degree programs. If the Department of Education is giving its imprimatur to this type of research, and policy is being based on it, then we are in real trouble.

My goodness! DL via video is archaic to say the least. This study had to have been funded by a teacher’s union.

This research is flawed from the very beginning. It is a common mistake that a traditional faculty member makes in thinking that the online course should mirror the f2f class. Watching lectures online is probably the worst example of pedagogy one could ever implement … therefore, the “slightly worse” result is due to poor instruction … not online learning.

It boggles my mind that the authors of this study consider it the “best-quality evidence that is available.” There is a ton of research out there that is structured almost identically to this study (going by the way it is presented in this brief analysis, at least). As jabberwocky12 said, the overwhelming conclusion is that there is “no significant difference.” It even has a name – the “no significant difference phenomenon.” Google it, and you’ll find many hits.However there is a fundamental flaw with most of this research. Care is generally taken to ensure that the two teaching modalities match each other as closely as possible, e.g. in-person lectures are matched with online recorded lectures. And hey presto, students learn just as well in each case. (or one method gets an edge over the other, but this fluctuates by study. They’re generally not large effects, and they wash out in meta-analyses.) This should surprise nobody as the content and delivery are identical. Think about it – by what mechanisim would bum-on-chair in a lecture theater magically bring about some sort of benefit to the student?It is a far more interesting question to ask what you can do online that you cannot do in person, and vice-versa. Then use each modality for its strengths. If one of the values of learning with others is the ability to work collaboratively, or “bouncing off” other people’s ideas in real time, then do that. If one of the benefits of online learning is using highly interactive activities, then do that. These are just examples, as there are a huge number of face to face and online pedagogies that will be suitable to different teaching contexts in different disciplines. Comparisons will be harder and conclusions will be more tentative, but then again education is a very complex process and distilling it into a “pure” research design is simply not going to tell us very much. Other research methods are required.

If this means video only online learning is compared to stand and deliver lectures then the results are only meaningful to service courses with huge sections or outdated lecture dominant courses.

This is simply an example of poor reporting by Ms. Li. Check out the better written article at Inside Higher Education. Figlio’s study validated 80 years of media comparison studies by showing “no significant difference” among the students as a whole (only Hispanics showed a difference). The silliness of equating an interactive online course with watching videos online has been noted already. Of course the basic flaw of this kind of study is that it assumes that all face-to-face are of similar enough quality with each other that the differences among them are greater than the differences between them and online classes (and vice versa).

The Chronicle does too much of this “reporting” that mirrors the McNews on television in which someone puts out a press release and the news outlet basically repeats it. Didn’t anyone at CHE even stop to think about the discrepancy between the headline and the content of the article?

Go to http://www.nosignificantdifference.org/ for additional “research” on the topic.

Thanks for the comments. Though the report cited here talks about online instruction, the main variable of the experiment seemed to be video lectures versus live ones. The story has been updated to reflect that.

In the link above, the paper this study is referring to is summarized here. I, for one, will be reading its entirety if I can locate it on my university’s digital library.Is it Live or is it Internet? Experimental Estimates of the Effects of Online Instruction on Student LearningDavid N. Figlio, Mark Rush, Lu YinNBER Working Paper No. 16089Issued in June 2010NBER Program(s): CH ED This paper presents the first experimental evidence on the effects of live versus internet media of instruction. Students in a large introductory microeconomics course at a major research university were randomly assigned to live lectures versus watching these same lectures in an internet setting, where all other factors (e.g., instruction, supplemental materials) were the same. Counter to the conclusions drawn by a recent U.S. Department of Education meta-analysis of non-experimental analyses of internet instruction in higher education, we find modest evidence that live-only instruction dominates internet instruction. These results are particularly strong for Hispanic students, male students, and lower-achieving students. We also provide suggestions for future experimentation in other settings.jeffreyyoung,Thanks for changing the article and providing a link for the source material’s abstract.CB-10,The questions regarding the instruction we raised are answered in the abstract: the videos were the same as the presentation given.OF COURSE THE LIVE PRESENTATION WOULD BE SUPERIOR! And of course universities expecting this to serve as the entirety of instruction for an online course would be doing their students a great disservice. Online classes typically take anywhere from 40-60 hours of preparation before beginning the class. That preparation should, ideally, mean creating content specifically for the instruction of the online class that would best meet the needs of the online learner.

Thanks for updating the article. But when the Chronicle changes the content of an article on its website in a substantive way, shouldn’t it note that it has done so in the body of the article itself? That is standard journalistic practice, and if you don’t regularly do that, you should.

Online education is fake education. I will never, never, never see wholly online education as real education or real school. Online courses should be required to marked with an asterisk, and online degrees should be legally required to be indicated as such on resumes. Let’s cut the joke out and be honest. There is no teaching, just self study. We don’t know who is taking the exams. The classes continually give credit for homework or discussion postings, but discussion posting are not a big learning experience, and we don’t know who is actually doing either the postings or the homework. It is all “pretend” school, just a convenience, and if the state legislators ever figure out what they can do, they will shut down all schools but one in each state and let everyone study online to save money. It’s not real, people. Purely online classes have no standards at all: they cannot have them, not so long as there is no real teaching, no real answering questions, no real demonstrating things on the board, no real active class discussion, no real exams where they are closed book and you know who is taking them. It’a all a fake.

Softshellcrab “will never, never, never see wholly online education as real education or real school” for the simple reason that he or she obviously has made up his or her mind and he or she is never going to be confused by the evidence. A generalization like this is wonderful. By the same logic, telephone conversations are not real, nor are email communications. Come to that, this isn’t a real discussion, since we’re not face to face. For that matter, even written letters aren’t real communication, because the sender has no way of knowing who’s going to open the envelop, and the recipient has no way of knowing who really wrote and mailed the letter. So much for distance communication and distance learning and teaching. Especially since classroom and lecture hall teaching is completely immune from substitute attendence. . . .

This is as clear a case of incompetence as I have seen in awhile. For Mr. Figlio to equate online instruction (his wording, he can’t escape it) with video lectures is an appalling display of ignorance. It seems clear that Mr. Figlio has never constructed, managed, or even participated in a well-designed virtual learning community. One would think this lack of experience would disqualify him from having the expertise to lead such research. Apparently not. Ignore it and move on. More good research is needed and welcomed.

“This paper presents the first experimental evidence on the effects of live versus internet media of instruction.” Not only does it not present the “first experimental evidence” (they are a few hundred studies too late for that), but all it does is compare a camera’s eye view of a classroom with a student’s eye view. It does not contradict or nor does it even add understanding to the USDOE meta-analysis, since it focuses on a single delivery mechanism (lecture capture), which is only one of many components to an online course. Perhaps our colleagues should consult with Dr. Thomas Gibbons, Dean of Northwestern’s School of Continuing Studies, who answers the question, Many people think of correspondence courses or taped lectures when they hear the words distance learning. Is that a fair description? “Absolutely not. That misses the important elements of group and faculty/student interaction.” http://www.northwestern.edu/newscenter/stories/2010/02/distance.html.

This is the type of “research” that gives graduate education a bad name. It exudes a poor grasp of what online and blended learning is and is becoming. And it does not share the rigor of the earlier DoE research reviews on blended, online and direct learning. Unfortunately, the reporter jumped too soon. doing your Homework is good!

I read the headline and when I clicked to the article, I thought I had misread. I fully agree with all of those who stated that on-line learning is not the same as video lecturing. The confusion of these terms shows a real lack of understanding of the basics of distance education. This research is worthless on the topic of the effectiveness of distance ed.

I am dismayed at the headline the Chronicle gave to this article on the Academe Today page, “Online Learning May Slightly Hurt Student Performance.” One realizes this is a gross mischaracterization of online learning only if s/he follows the link to the actual article. The “video lecture” does not characterize online learning, and this is well documented. I expect variation in quality of research, but I expect consistent, higher quality from the Chronicle.

catkelley,Thanks for making your last comment. If, indeed, two of the authors are students (as I am of instructional design), and given it is clear this is an unpublished paper (something I had missed even though I read the abstract), I wholeheartedly offer my condolences to the writers of this paper.Even then, however, I have to say, how could anyone in their right mind think such a limited study would overturn, or even bat an eye in regards to, the conclusions drawn from a meta-analysis of over 1000 (spanning over twelve years of time) studies conducted regarding online instruction? This, too, is still not really given credence in the writing of this article.I agree with you in that the Chronicle has made a bad call – one hurtful the many educators who work in an online or hybrid environment, making a REAL difference in REAL lives every day. Man, that comment from ianative regarding softshellcrab is a keeper! Well said!

Of course it does. There isn’t any real-time interaction and without that there is no real-time feedback from the professor. I would like to see the research that says it benefits Hispanic students. Are they looking at sub-groups of Hispanic students and what are they considering Hispanic? Perhaps those who are experiencing language barriers benefit from the video lecture because they can rewind the video to make sure they understood the professor. I work at a HSI and my student complain about lectures that are given via interactive tv and lectures that are taped on video. Nothing beats real-time interaction and feedback.

The full paper is far more nuanced and tentative than what the above review makes it seem. The authors do NOT equate watching video recordings with online learning. On the contrary, they clearly explain that the lectures were integrated in a rich online learning environment. Moreover, they candidly list external factors that may have influenced the results: in particular, that it was possible for students of the group meant to only attend the lectures face to face, in whose accounts access to their video recordings had been blocked, to by-pass this block by using someone else’s account, thus benefitting from both the F2F and the online version – whereas students meant to view the videos online where effectively barred from attending the lectures.The authors also state the restrictions imposed by the university as to the sample size and the number of times students who had volunteered could be contacted.They are very honestly tentative in their hypotheses about the slightly lower exam performance of students who only viewed the lectures online. One of them is that the possibility of viewing videos at any time may have lead some students of the “online only” group to attempt to cram this viewing until shortly before the exams, which did not work (while you can cram reading texts, studying a video recording takes more time). They are even more tentative in their hypotheses about why some subgroups of the “online only” group seemed more affected: for people who are not native speakers of the language used in a video, studying it for an exam takes even more time. They therefore call for a duplication of the experiment in other universities where there the proportion of students from less advantaged backgrounds, and/or where there is not the same rich online learning environment.The full paper clearly shows that while the authors do not equate at all e-learning with viewing lectures in online recordings, their worry is that some universities – particularly those with limited financial means – might limit their (e-learning) offer to online recordings of lectures, without providing the same rich online learning environment students are able to use in the university where they did the research, and without measures to counter the temptation to wait until too late to study these online video recordings. The authors’ own proposal of such a counter measure, i.e. to force downloading and viewing of the videos within some days of the lecture, may be arguable, sure. There are other more constructive alternatives, e.g. asking students to take turns in posting their notes on a given lecture in a wiki for the course. Finally: I do not know the authors of this paper. I was prompted to read the full paper by a jarring note in the above review, and by the outpour of damning comments that are apparently only based on this review, and the misunderstanding it fosters, with a few welcome exceptions where the commentators went beyond the impression this review gives: gabrielledean’s comment 18, do_the_right_thing’s comment 47.Granted, the authors should perhaps have worded their abstract so as to more correctly reflect the tentativeness of the full paper and context of the research. This might have prevented skewed reviews like the one above, which remains skewed in spite of update mentioned in the Editor’s note at the end. However, one shouldn’t damn a research in comments that may remain online forever, and thus risk harming the credibility of the researchers, on the sole basis of a review, even if it is published in the Chronicle of Higher Education, and even if the abstract itself on which the journalist apparently based her review may have been misleading.

calmansi & ardvaark: My problems with the study are not primarily with the conclusions or methodoloy, though as I note in my first comment above I am not a fan of this approach. Still, many have taken this approach, and I cannot fault them for taking the same approach to this issue that so many others have already done. And I have no doubt that as you say, they are more nuanced in their interpretation in their full paper. I have not read it, because I don’t want to spend $5 for yet another paper comparing online vs. face to face instruction. And indeed this is what _does_ annoy me about the paper – it is “yet another” such study. Yet these authors call this the “first” experimental study comparing online with face to face instruction. This assertion is the very first sentence in their abstract, so it is really difficult to imagine that the full paper demonstrates that they are aware of the vast research literature in this area. I find it mind-boggling that they would not be able to find the decades worth of research in this area in even a very cursory literature review. And more to the point, as I said in my second comment above, I cannot fathom why the Chronicle would publicize this work, which is (as you say) quite preliminary, by the authors’ own assertion. It has not been peer reviewed, and is only available as a tech report that most of us would have to pay for, thus preventing most of us from fact-checkin what the Chronicle has written. And as you also note, the Chronicle article (especially the original version) did not go into the nuances, but blithely asserted that online learning harmed student performance, especially that of Hispanic students. My beef is primarily with the Chronicle. I think the authors of the study should be taken to task for not understanding their research area, however – they most certainly seem unaware of the other research that has been done in this area.

@calmansi & @ardvaark: note that this aricle (even its title) and the paper itself have been altered since this story went live. The paradox pointed out did not originally exist.

If the posts above that are critiques of the quality of the research design were submitted by students in my research methods course, there might be a few Bs, but only #18 would get a clear A for demonstrating an understanding of the logic of conducting an experiment with random assignment of subjects to conditions. On the other hand, perhaps the study authors and certainly Sophia Li do not seem to understand how rigorous design constrains the conclusions that can be drawn from the results.

@grandeped (comments 57/8): when an article is modified and there is an editor’s note that says: “This article was updated on 06/22 to more accurately characterize the research”, it can be presumed that the original version was misleading. So re the quote from the authors that was removed and that is still reported in (http://www.edugeekjournal.com/2010/06/21/for-the-one-millionth-time-this-is-not-online-learning/)one may safely assume that the journalist had initially misreported/misunderstood what the person had said.@ rpm13 (comment 59): very intresting: could you develop the part about rigorous design, please? @ ardvaark: there are human teachers in online teaching too, though. Teachers’ ability to facilitate students’ learning processes is in great part a matter of teacher/students ratio, whether online or F2F. In a F2F course with 200 students, that’s not easy either. Sometimes, students actually feel more at ease contacting their teacher/tutor about a problem in an online course than in a F2F course.However, the point is that the conclusions of the study reviewed by the Chronicle are not about online teaching in general, but only about a – hopefully small – part of it: recorded lectures. In fact, the authors are not globally against online teaching – nor even against recorded lectures, for that matter. They warn of possible issues with recorded lectures when students do not know how to use them efficiently for learning.

@calmansi – I have worked in journalism media before, and you can’t assume that something is removed just because it is misleading. That is not a safe assumption. There are many reasons articles are updated. There is also a good chance that this quote was copied and pasted from either an early version of the report itself or an email correspondence (that is how most interviews are now conducted). Even if not, I don’t see how you can misrepresent someone saying that they have the highest or best quality evidence to date.My overall point being that you and others are critiquing many of the commenter’s critiques, which is very difficult to do now that everything has been changed and updated. Many of the comments, including my original one (#2), were to the original Chronicle article that was posted. The so-called “hysterical” defenses were reaction to an article (not necessarily the study) that claimed that the best evidence out there might indicate that online learning is slightly less effective than face to face learning. The original article has been amended to take away this extreme edge, so many of the comments, especially the earliest ones, should be taken in that light.

@ Raza Khan: the experiment this study reports took place in a single 3-months course in micro-economics, and lasted for the whole course. The researchers did not “lump different types of students” either: but they also took into account their backgrounds as a possible variable.