• July 31, 2014

A 'Stealth Assessment' Turns to Video Games to Measure Thinking Skills

Avoiding complications like test anxiety is one benefit of a method for evaluating difficult-to-capture abilities

A 'Stealth Assessment' Turns to Video Games to Measure Thinking Skills 1

Sarah Kiewel for The Chronicle

This computer game, ostensibly about ecology, can actually measure a variety of critical-thinking skills in students, says Valerie J. Shute, a professor at Florida State U., who is playing the game on her laptop.

Colleges no longer simply want to know what their students know, but how they think.

Higher-order thinking skills are "something that schools are paying a little bit more attention to these days," says Jeffrey Steedle, a measurement scientist at the Council for Aid to Education, whose Collegiate Learning Assessment essays are used at several hundred colleges to test students' abilities to synthesize arguments and write persuasively. "It's largely in response to the recognition that these skills are needed to be competitive in the global marketplace."

But educators also say that paper-and-pencil examinations have limits—for one thing, knowing that you are being tested can drag down performance—and they are looking for new methods to measure skills like critical thinking, creativity, and persistence.

Valerie J. Shute, an associate professor of educational psychology and learning systems at Florida State University, believes she has a solution in "stealth assessment"—the administering of tests without students' knowing.

To do that, Ms. Shute and other stealth-assessment researchers have turned to video games, which let educators watch students solve complex tasks while immersed in virtual worlds. How students react to new challenges and put evidence together—without the pressure of test proctors breathing down their necks—can reveal a lot about creative problem-solving skills that traditional testing cannot deliver. "A lot of important stuff happens when playing games," Ms. Shute said. "You're just doing. You're in the process."

Ms. Shute, who first studied stealth assessment with a video game for undergraduate students more than 20 years ago, sees applications for students of all ages. She is now helping two of her graduate researchers test the technology with sixth graders in Florida, And she is already looking for new ways to use the technique in her graduate courses.

"Everybody likes to play," she says. "And so much could be done using games."

Matthew Ventura, an associate research scientist at the Educational Testing Service who worked on stealth-assessment research with Ms. Shute, says the technique will be especially helpful in eliminating test anxiety, which can hurt students' performance. That and its potential to test intangibles like creativity and problem solving, he said, make the technique attractive to educators and video-game developers alike.

Virtual Worlds, Real Skills

Educators have long believed that video games and virtual worlds could be used to supplement classroom instruction, although not necessarily as testing tools.

In 1986, when Ms. Shute was a postdoctoral fellow at the Learning Research and Development Center at the University of Pittsburgh, she designed a computer game to teach undergraduates the principles of microeconomics. The game was set in the virtual farming community of Smithtown, where students could play around with different economic factors—like product price, labor costs, and population size—to see how they affected the market. A student who decided to raise the price of coffee, for instance, would see demand for his or her product fall as a result. The software also let students plug in hypotheses about the outcome of their experiments.

On its surface, the game was a way for students to get acquainted with basic economic ideas—and maybe have a little fun in the process.

But it was also teaching students a deeper lesson about scientific inquiry. If they changed too many inputs at once, they found it difficult to determine what caused a sudden change in the market, and had to go back and experiment with the inputs to determine exactly what was happening.

The question that grew out of Smithtown, Ms. Shute says, was, "Wouldn't it be lovely to actually pass along the log files of what students did in order to look at their scientific-inquiry skills?"

Since then, Ms. Shute has been working on a framework to help educators design and execute stealth assessment.

She looks first to the core competencies—critical thinking, empathy, persistence—that she wants to test, then breaks them down into smaller goals. She can then tie those theoretical skills to actual tasks in a video game. For instance, an instructor looking to test a student's grasp of systems thinking—understanding the complex relationships among parts of a whole—might ask players to complete tasks that show information gathering, developing hypotheses, and tracing causal relationships.

Rather than conducting a one-off exam, Ms. Shute says, continuing stealth assessment "enables you to have a very systematic representation of the stuff you're interested in."

But for Ms. Shute, stealth assessment is not just about gathering data. It's also about improving teaching.

If instructors know where students need the most help, they can quickly tweak their courses—and their games—to make up for those deficiencies. Students who need help developing critical-thinking skills, for instance, may be asked to repeat a level or to take on additional tasks in the game until their performance is satisfactory—all without interrupting their play. "The idea of stealth assessment is really to make it merge into the fabric of the learning environment," Ms. Shute says. "My goal is to blur the distinction between learning and assessment."

Crunching all the data on student performance and tweaking computer games were major tasks in the days of Smithtown, but today's technology is making it possible to test increasingly complex critical thinking using virtual worlds.

Taiga Park, a computer game developed by the University of Indiana's Center for Research on Learning & Technology, is one of Ms. Shute's favorite vehicles for stealth assessment. On its surface a game about ecology, Taiga Park requires players to look for the cause of a widespread fish die-off in a virtual river by "interviewing" park rangers, environmental scientists, and the owners of a logging company. While students learn about pH levels and runoff, they also come away with lessons on data analysis, complex cause-and-effect relationships, and communication.

With the stealth-assessment framework behind it, Oktay Donmez and Yoon Jeon Kim, two of Ms. Shute's graduate students at Florida State, are planning to test Taiga Park with about 50 sixth graders. Starting in the spring, the researchers will study the students' performance data from the game to see how effective stealth assessment is at measuring their complex thinking.

"My goal is not only to design, develop, and implement stealth assessment within a game, but also to test its effectiveness," Mr. Donmez says.

Although much of the research in the field—especially when dealing with elementary and secondary students—is centered on video games, Ms. Shute is quick to point out that stealth assessment is widely applicable as a tool, not just a game.

Working with Florida State graduate students in a course called "How to Write Excellent Literature Reviews," Ms. Shute found that she could use routine assignments—like peer reviews and summaries of research material—to analyze her students' higher-order thinking skills. All assignments can be linked back to a larger skill, she says. "Evidence is everywhere."

Ms. Shute also hopes that stealth assessment might engage students unmoved by traditional teaching and testing.

"We have this whole group of kids who are not engaged with school, and appropriately so, because schools are so antiquated," she says.

With field research on stealth assessment still ahead of them, both Ms. Shute and her students are optimistic about the technology, especially for a generation of students who grew up with video games at home.

"They're going to play video games anyway," Ms. Kim says. "We can actually embed something that can help them learn."

Comments

1. sitepsu - November 12, 2010 at 09:33 am

Wow this is intriguing. I'd be interested in what the folks involved in Penn State's Educational Gaming Commons (is that the correct name?) think about this. Could we consider co-sponsoring a forum to discuss this?

2. rginzberg - November 12, 2010 at 01:43 pm

I wonder how this squares with the observation that non-traditional students constitute ever increasing portions of college populations?

Does the 45 year old returning adult student do as well on skill tests that are measured via video games as the 18-22 year old student? Has this even been investigated?

3. midevilprof - November 12, 2010 at 04:02 pm

What about essay exams that ask questions aimed at critical thinking and problem solving? Some of us have been using those for years, insted of or in addition to bubble-sheet style tests.

And how easily is it to work up similar games for different fields? Could someone create a game for historical understanding or literary criticism? If so, I'd sign right up!

4. arrive2__net - November 13, 2010 at 03:54 am

Using game technology to measure higher order creativity and thinking skills does seem like a fundamental advance. Combining learning with formative assessment in an electronic game environment seems to have tremendous promise, however it may be necessary to demonstrate that the skills transfer into "real world" situations where learning and performance may not be as well structured, and evaluation does take place. Perhaps student could develop advanced skills, but may also have to learn to transfer them to practical applications outside the artificial cyber-world.

I have to wonder about the ethical implications of "stealth assessment".

By the way that's "Indiana University", not University of Indiana.

Bernard Schuster
Arrive2.net

5. shenrikson001 - November 14, 2010 at 03:13 pm

I am currently enrolled as a student at Walden University in an online Masters Program; Instructional Design Technology, which has an emphasis on online performance improvement. I am responding to this article as part of an assignment and would appreciate any feedback.

I found this article very intriguing regarding the ability to measure the students higher order creativity and thinking skills. I've recently retired from the Navy and for a while was aware of the development of computer gaming that provided sailors basic and advanced geolocation theories and training while immeresed in a gaming environment, http://www.invism.com/projects/software-applications/geocommander/geocommander.html. This program has proven to be very successful and has expanded to other areas of military training. At a time when training budgets are being axed, this type of training is a very budget friendly way to make learning better.

I think the answer to midevilprof's question is why not....if it can be taught in a class room, it can be taught on a computer and with the way that our children are growing up on computers today, the classrooms of today will not be able to keep their attention with out some kind of electronic stimulus. We brought it on ourselves by buying electronic games and toys for our kids, we might as well make the most of it.

To Bernard, I can understand your concerns regarding the ethical aspects of "stealth assessment", and which I agree is a very legitimate concern. It would be interesting to see how they explain the process to the group of six graders and their parents
when this paradigm shift of educating their children is presented to them for what seems to be an isolated assignment. If my child were in that group and it was communicated to me what the "grading" scale was going to be, how the process would be tweaked to better complement my childs learning abilities, how and when the assessments would take place, what the roadmap for advancement to the next level was, and where my child was in comparison to children following a more typical educational path, and this all happened with my child having being aware of the overall assessment and training picture but immune to the fact that the assessments were ongoing I think I would be onboard with that.

6. shenrikson001 - November 14, 2010 at 03:17 pm

I guess I should have at least signed my comment:

Scott Henrikson
scott.henrikson@waldenu.edu
http://shenrikson001.wordpress.com/

Again, I would appreciate any comments or feedback on my response.

Add Your Comment

Commenting is closed.

subscribe today

Get the insight you need for success in academe.