Assessment Takes Center Stage in Online Learning

Distance educators see the need to prove that they teach effectively

By DAN CARNEVALE

Students at Western Governors University aren't required to take any courses. To earn a degree,

ALSO SEE: Three sample assessment questions from Western Governors University in the area of quantitative reasoning, and the answers.

they must pass a series of assessment exams. The faculty members don't teach anything, at least not in the traditional sense. Instead, they serve as mentors, figuring out what students already know and what courses they need to take to pass the exams.

Assessment also plays a big role at the University of Phoenix Online. In a system modeled after the university's highly successful classroom offerings, students are grouped together in courses throughout an entire degree program, and they are given batteries of exams both before and after the program. The tests enable the university to measure exactly how much the students have learned, and to evaluate the courses.

Indeed, assessment is taking center stage as online educators experiment with new ways of teaching and proving that they're teaching effectively.

And traditional institutions, some observers say, should start taking notes.

Education researchers caution that distance educators are still in the process of proving that they can accurately assess anything, and that comparatively few distance-education programs are actually participating in the development of new testing strategies.

One difference between assessment in classrooms and in distance education is that distance-education programs are largely geared toward students who are already in the workforce, which often involves learning by doing. In many of the programs, students complete projects to show they not only understand what they've learned but also can apply it -- a focus of many assessment policies.

In addition to such projects, standardized tests are a key part of assessments in distance education. These tests are usually administered online in proctored environments, such as in a student's hometown community college.

Western Governors and the University of Phoenix Online are among the most visible institutions creating assessment methods, but they are not alone. Many other distance-education programs use some form of outcomes-based assessment tests, including Excelsior College (formerly Regents College), in Albany, N.Y.; Pennsylvania State University's World Campus; Thomas Edison State College, in Trenton, N.J.; the State University of New York's Empire State College; and University of Maryland University College.

All of higher education is moving toward outcomes-based assessments, with online education leading the way, says Peter Ewell, senior associate at the National Center for Higher Education Management Systems. The push for new assessment models in online education comes largely from competition with its older brother, traditional education, says Mr. Ewell. Because distance education is comparatively new, he says, critics often hold it to a higher standard than traditional education when judging quality. It has more to prove, and is trying to use assessments that show its effectiveness as the proof.

Online education is only one of several influences putting pressure on traditional education to do more to assess the quality of courses. Accreditation agencies, state governments, and policy boards are all heading toward an inevitable question, Mr. Ewell says: How much bang for the buck is higher education putting out?

But Perry Robinson, deputy director of higher education at the American Federation of Teachers, says assessment exams shift the emphasis away from what he considers the most important element of learning: student interaction with professors in a classroom.

The federation has been critical of distance learning in the past, saying an undergraduate degree should always include a face-to-face component. Mr. Perry says having degrees that rely on students' passing tests reduces higher education to nothing more than job training.

Also, Mr. Perry doesn't want to see the role of the professor diminished, because that person knows the material the best and works with the students day after day. "Assessment is involved in the classroom when you engage the students and see the look of befuddlement on their faces," he says.

But Peggy L. Maki, director of assessment at the American Association for Higher Education believes that all of higher education will move toward a system of assessing outcomes for students. Although distance education is contributing to this movement, it isn't the biggest factor, she says. "We're talking about a cultural change."

Some of this change is prompted by the demands of legislators and other policy makers, Ms. Maki says. Also, institutions are feeling pressure from peers to create outcomes-assessment models. "I think there have been more challenges with people saying, 'Can you really do this?'" she says. "When they do, others say, 'Well, we better follow suit.'"

But traditional and distance-education institutions alike are struggling to figure out how to use the the results of assessment examinations to create programs and even budgets. "This is the hardest part of the assessment process -- how you use the results," Ms. Maki says.

Western Governors University's assessment system is intended to measure the students' competency in specific subjects. Because it doesn't matter to W.G.U. whether the students learned the material on their own or from courses they've taken through the university, the entire degree revolves around the assessment tests.

The university doesn't create its own courses. Instead, it forms partnerships with other universities around the country that have created online courses in various subjects. A student seeking a degree must show competency in a number of "domains." These include general education, such as writing and mathematics, and domains specific to the subject, such as business management.

Western Governors officials create some of their own assessment examinations and buy some from other organizations, such as the ACT and the Educational Testing Service.

For W.G.U.'s own exams, experts from the professional and academic arenas collaborate to determine what students need to demonstrate to prove they are competent in a field. Unlike traditional colleges, Western Governors separates assessment from learning. The professors who grade the assessment exams have not had any prior interaction with the student.

Some of the tests rely on essays or multiple-choice questions, while others are projects in which students must show they know how to use their knowledge. Completion of a degree involves a student taking versions of all these exams.

Projects are designed to simulate tasks that an employee might undertake in a real job. For an associate degree in applied science, information technology, and network administration, for example, students plan computer networks for small companies. The students present this plan in a five-page essay that details technical and financial requirements for enacting the plan, similar to a report an employee would present to a manager at such a company.

Each major has its own project requirements. For example, in an associate degree of business science, students have to create a marketing plan for a small company. The project takes the form of a five-page essay in which students detail how a company can improve its reach to customers.

These projects are part of the university's portfolio requirement. Professors review the portfolio to determine whether the student has demonstrated the knowledge and skills to earn a degree. "Our degrees do ensure an appropriate level of problem-solving skills," says Alec M. Testa, director of assessment at the university.

Mr. Testa says that Western Governors learned how to assess outcomes from other institutions, such as Excelsior College and Alverno College, a women's institution in Milwaukee.

The institution has found that assessment is particularly popular with private companies whose employees take the university's courses. The university gets much of its money as donations from such companies, and it works with them to create degree programs and to find out what knowledge students must have to be able to do jobs at the companies well.

Peggi Munkittrick, senior director for teaching and learning strategy at SCT Corporation, a technology company, says having students work on projects instead of listen to lectures is an effective way to learn. She says Western Governors' courses often have students work in groups to solve problems, as they would in a company. "They actually have to use the skills of working in a collaborative fashion to create a product," she says.

The market will determine how fast outcomes assessment creeps into traditional education, says Utah Gov. Mike Leavitt, a Republican who was one of the founders of W.G.U. "If competency-based degrees become attractive, you could see universities doing both," he says.

Western Governors' assessment model also holds institutions accountable for the quality of their programs, he says. "You don't measure inputs," he says. "You measure outputs."

The University of Phoenix took the assessment practices of its face-to-face program and expanded them to its fast-growing online program.

Like their counterparts in the university's classrooms, students at the University of Phoenix Online take the same test just before they begin an academic program, which typically lasts at least two years, and again at the end. The university can then analyze the data to see how much the students have learned from the program, providing a type of quality check on the courses.

The tests usually consist of about 100 multiple-choice questions. University officials say the models have helped the university affirm the effectiveness of new pedagogical strategies that officials there have found work well both online and on campus.

University of Phoenix programs focus on specialized training for adult students, but also include general education. Even in, say, a business-management program, students are exposed to general courses, such as history, just as a traditional student would be.

"Theory and practice are emphasized, as well as individual development," says Brian Mueller, chief operating officer and vice president of the university.

The University of Phoenix also works closely with private companies to develop its programs. The institution finds out what skills different companies want from their prospective employees, and then works with professors who specialize in the field to develop a curriculum. The faculty members, who are called "facilitators," teach courses from this prepared curriculum.

Through the tests and projects, the university assesses a student's cognitive abilities, critical thinking, and communication skills. Faculty members also provide peer evaluations of each other's work to help improve teaching skills. "We measure everything that moves," says Karen Spahn, director of institutional research at the university.

"The higher-education community continues to be challenged by the assessment of student outcomes," says Charles M. Cook, the director of the New England Association of Schools and Colleges' Commission on Institutions of Higher Education.

Traditional education, he says, will probably slowly follow the lead of distance education. "It's in a better position to assess student learning than a traditional establishment," Mr. Cook says. "It's truly more student-focused and outcomes-focused."

Arthur E. Levine, president of Teachers College of Columbia University, says that institutions are still in the beginning stages of developing these assessment models, but that they will eventually be used in all aspects of higher education. "It's the future of both traditional and nontraditional education," he says.

As new ways of teaching and new evaluation techniques are invented, the old process of accumulating credits to finish a degree will eventually become obsolete, Mr. Levine says. "The process base was a natural consequence of the Industrial Revolution," he says. "But now we're entering the information age."

Lee R. Alley, manager of instructional-technology projects at Montgomery College, in Rockville, Md., and Kathryn E. Jansak, associate provost of Shawnee State University, have found that some of the better online programs use outcomes assessment (http://www.worldclassstrategies.com/papers/ keys.htm). "It turns out that quality assurance and quality assessment line up on top of each other in a very tidy sort of way," says Mr. Alley.

He believes that distance education will influence traditional education as professors and administrators learn what experiments in distance education work. "It's turning out to be the R&D lab of higher education," he says. "Those principles of learning science pretty much apply in any domain."

Although online education may be taking a leadership role in assessing student outcomes, not that many distance-education programs rely on assessment models.

In Mr. Alley's view, neither traditional education nor distance education is doing enough to experiment with new teaching techniques and assessment. "We don't have enough leaders," he says.

All the regional accreditation agencies have been working on a list of "best practices" to serve as guidelines for institutions building online programs.

A report on the guidelines project strongly urged institutions to evaluate the effectiveness of their programs by assessing how well students did and how much they learned. The report recommends assessing "the extent to which student learning matches intended outcomes, including for degree programs both the goals of general education and the objectives of the major."

The report also suggests determining whether the degree program include "measures of student competence in fundamental skills such as communication, comprehension, and analysis." The report does not give specific advice on how the institutions should assess these aspects.

The online programs are competing for some of the same students traditional institutions want. Students who are worried about whether they'll get jobs are going to be asking for evidence that one college program is better than another, says Mr. Ewell, of the National Center for Higher Education Management Systems.>

"Those competitive pressures are being felt in traditional education because students are starting to ask those questions," he says. "The effect is going to be measured in years, not months."

1. You operate a machine that stamps bottle caps out of 3-inch-by-3-inch aluminum squares. Occasionally, the machine produces an unusable cap, a reject, that must be recycled. The number of rejects made at different production rates is shown in the graph below. Today you have been told to produce 600 caps per hour. Approximately how many caps total should you have to produce to end up with your quota of 2,400 good ones? (The coordinates of the points on the graph are, from left to right: (75,4), (150,6), (300,12), (600,24), and (1200,50).)

a) 2400 b) 2448 c) 2522 d) 2548 e) 2616

2. You can install 12 square yards of flooring in 3 hours and 15 minutes. You need to put the same type of flooring in a room that is 10 feet 4 inches by 15 feet 9 inches and figured it would take you 39 hours and 33 minutes. Just before starting, you do a quick estimate in your head and think you might have made a mistake. What error, if any, did you make?

a) accidentally doubled the time needed to install 12 square yards.
b) didn't make any mistakes -- the answer was right.
c) rounded the room measurements to the nearest foot.
d) didn't convert the room measurements to square yards.

3. Psychologists tell us that the average person dreams about five times each night. Many people have experienced "precognitive" or "predictive" dreams sometime in their lives in which many details of a particular dream later turn out to come true. Some people claim that the fact that many people have had predictive dreams of this kind proves that our minds have the power to foretell the future. Explain why, based on probability theory, it is reasonable to expect people to have occasional predictive dreams without needing to invoke any unusual mental powers.

1. The problem first requires you to know exactly what is being asked. This might be re-phrased: "how many caps in addition to the 2,400 that I need must be produced in order to cover the number of rejects at that particular production rate?" By looking at the graph carefully, you can see that at 600 caps/hour, 24 rejects are produced out of every 500 caps -- or 476 good caps per 500 are produced at that rate. Applying the resulting proportion of 500/476 to 2400 yields 2521 with a remainder; since you can't have a remainder of a cap, the correct choice is 2522 (choice c.).

2. The problem first requires a determination of whether the calculation is wrong. Here a quick estimate will do. For instance, a 12-square-yard room might be one that is 3 yards by 4 yards -- or 9 feet by 12 feet. The room to be floored is less than twice this big, so an appropriate time estimate might be about five hours. The answer arrived at was eight or nine times this amount, so it is clearly wrong. That eliminates b. So what went wrong? Answer c. doesn't nearly account for how big the difference is. Nor does b. Mistake d. looks good because the difference between square yards and square feet is nine -- or about the amount that the original estimate is wrong.

3. A good response here will examine the approximate number of "trials" (in this case individual dreams) that an average person may have over a given period -- say 5 or 10 years. A minimal response would simply comment on the raw probability of a given person experiencing at least one "predictive" dream during this time period. A better response would consider the joint probability of at least one of these many dreams corresponding in some way to an actual event -- which would require setting up a relative probability in the other direction. A really good answer would use both of those estimates to arrive at a joint probability estimate -- and would discuss the plausibility of all three elements of the calculation.

Easy-to-print version

E-mail this article