From the issue dated October 15, 1999

Sociology, Not Engineering, May Explain Our Vulnerability to Technological Disaster In studying meltdowns, plane crashes, and chemical leaks, scholars blame high-risk cultures, unpredictability, and even 'safety' measures By D.W. MILLER Bhopal. Challenger. Chernobyl. Exxon Valdez. ValuJet. Faced with modern technological catastrophes, we console ourselves that really bad ALSO SEE: For a Scholar of 'Normal Accidents,' Risky Activities Bring Danger and Data accidents are rare and their causes are knowable: Design flaws ... operator incompetence ... corporate greed. Whatever we can name, we can fix -- with better, safer devices and procedures that anticipate an ever-wider range of human error. Perhaps nothing has done more to turn that comforting thought inside out than a 20-year-old idea by Charles Perrow, a Yale University sociologist. An organizational theorist, Mr. Perrow was asked to advise a Presidential commission that was investigating the 1979 accident at the Three Mile Island nuclear-power plant. Operators there narrowly averted a reactor meltdown after confronting a combination of small mishaps that was unique, unexpected, and therefore incomprehensible: a blocked valve here, an open drain there, a dangling tag that obscured a critical dial. Three Mile Island "was not due to equipment failure, or design failure, or operator failure, because no one could anticipate all this," he recalls thinking. "It was a 'normal accident' -- an accident built into the system." That epiphany led Mr. Perrow to develop the theory that some complex technologies can never be made safe. For a book-length project, he fleshed out his idea with examples from aviation, chemical manufacturing, marine transport, nuclear power, the space industry, and other areas. This month, Normal Accidents: Living With High-Risk Technologies (Basic Books, 1984) has been brought back into print by Princeton University Press, which is publishing a new edition with a postscript by the author. The book popularized the idea that catastrophic failures rarely have a single cause. And it helped transform the study of technological failure by showing how complex systems thwart our efforts to make them safe. The book's dark view of safety efforts seemed prescient in light of fatal disasters, such as the Challenger accident and Chernobyl, soon after its publication. Since then, Mr. Perrow's cautionary tone and ideas about systems failures have seeped into the scholarship of accident and risk not only in the social sciences, but also in finance, medicine, and engineering. Even those who disagree with his conclusions base their own reasoning on his framework. Mr. Perrow's theory rests on two basic notions. First, he says, some kinds of complex systems can never be made accident-free, because safety devices and other components interact in ways too varied for designers and operators to predict or understand. He calls that quality "interactive complexity." Second, in some systems, small errors can cascade so quickly that operators don't have enough time to figure out what's going wrong and make sound decisions to head off disaster. He calls that "tight coupling." "No matter how hard we try, no matter how much training, how many safety devices, planning, redundancies, buffers, alarms, bells and whistles we build into our systems," he has written, "those that are complexly interactive will find an occasion where the unexpected interaction of two or more failures defeats the training, the planning, and the design of safety devices." In fact, he argues, efforts to make a technology safer may actually make it less safe by increasing the odds of unpredictable interactions. Other scholars have added their own wrinkles to Mr. Perrow's efforts to replace the "name and blame" approach with a keener understanding of how "failure is built into the system." "His book talked about how operators are confused by technology, but not how their attitudes contribute to the accidents," says Diane Vaughan, a sociologist at Boston College and the author of The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA (University of Chicago Press, 1996). "I look at cognitive processes." After Congress investigated the Challenger explosion, she says, "the popular perception was that it was a result of production pressure and a decision by managers to launch even though the engineers warned against it." In fact, she concludes, the fault lay not with individual managers but with NASA's culture of conformity. The final decision to launch, she writes, was shaped by failures throughout the program. Key managers were not required to know that the shuttle had been designed for launching only in temperatures above 31 degrees. Engineers couldn't acknowledge that they didn't fully understand how the booster rockets performed in extreme cold. Contractors perceived that their reservations about launching had to be backed up with solid proof of the risks. When decision makers encountered conditions they had never before faced, she says, "it seemed logical and acceptable to continue with the launch." Scott Snook, a scholar of organizational behavior and an Army officer teaching at the United States Military Academy at West Point, makes a similar point in his new study of the accidental downing of two U.S. Army helicopters in 1994. In that incident, Air Force F-15's patrolling Iraqi airspace misidentified two helicopters on a U.N. observation mission, and an American surveillance aircraft failed to intervene. Twenty-six people died. In Friendly Fire: The Accidental Shootdown of Black Hawks Over Northern Iraq (forthcoming from Princeton), Mr. Snook concludes that the accident was partly the result of gradual deviations from established communication procedures, which the U.S. military had never modified to account for the helicopters' new role. Paradoxically, he says, "the more fixes you throw out there, the more people will break the rules to get the job done." Organizations in charge of risky technology sometimes overstate how prepared they are, says Lee Clarke, an associate professor of sociology at Rutgers University and a former student of Mr. Perrow. In Mr. Clarke's new book, Mission Improbable: Using Fantasy Documents to Tame Disaster (University of Chicago Press), he examines official disaster-response plans for oil spills, reactor meltdowns, and nuclear war that he considers absurdly unrealistic. Those plans are merely "fantasy documents," he says, symbolic "tools of persuasion" created by corporate and government planners to reassure employees, regulators, the public -- and themselves -- that they know just what to do in any emergency. Unfortunately, he writes, they often don't. For instance, the industry's plan for responding to a 200,000-barrel oil spill in Alaska's Prince William Sound assumed perfect weather conditions and promised the recovery of 95 per cent of any spillage. After the Exxon Valdez ran aground there in 1989, it became clear that planners had based their cheery scenario upon the conditions of a 4,000-barrel spill, because that was what they had experience with. The problem with fantasy documents, Mr. Clarke writes, is that they "can contribute to increased danger by decreasing vigilance and diminishing the capacity for organizational learning." Mr. Perrow's ideas are credited with sparking improvements in patient safety. David M. Gaba is an anesthesiologist and a professor at Stanford University's medical school who was originally trained as an engineer. He tries to look beyond technical, medical issues to understand how economic and legal pressures and the medical culture contribute to mistakes that harm patients. And, to train medical personnel in how to respond to crises caused by an unpredictable chain of factors, he has developed a portable "patient simulator," in which various emergencies are inflicted upon a dummy hooked up to hospital monitoring equipment. Of course, not all scholars of systems accidents are in the business of studying tragedy. Several researchers have been studying examples of what they call "high-reliability organizations," technological systems that have avoided catastrophic failure and even become safer over time. Todd La Porte, a political scientist at the University of California at Berkeley, and a team of fellow scholars spent about 10 years observing the operations of three types of entities that, he says, "were doing much better in terms of safety than they ought to have been": air-traffic control centers at airports in Oakland, Cal., and San Francisco, the nuclear-power plant at California's Diablo Canyon, and the aircraft carriers U.S.S. Enterprise and U.S.S. Carl Vinson. They discovered that officials of these entities minimized accidents by creating an intense "culture of safety." They were good at learning from mistakes, they delegated decision making in times of crisis to the personnel most familiar with the failing components, and they assigned redundant supervisors to oversee critical tasks, like monitoring radar screens. Mr. Perrow calls this an "optimistic" view of safety. High-reliability scholars believe, he writes in the new edition of Normal Accidents, "that if we only try harder we will have virtually accident-free systems even if they are complexly interactive and tightly coupled." Mr. La Porte, however, replies that his work is descriptive: "We're not saying, 'Here's how you do it.' We're saying, 'Here's the way it's done.'" The study of high-reliability organizations, he says, is not a theory of systems failure -- and it's not particularly optimistic, either. If their research shows anything, he says, it's that cultures of safety are difficult to achieve and difficult to maintain for a long time. Robert G. Bea, a civil engineer who left the oil and gas industry 10 years ago for a professorship at Berkeley, echoes that view. In a long-term study of safety in 15 petrochemical companies, he found only three success stories, and even they reverted to less-safe practices over time. He thinks that is because making whole systems safer requires frequent retraining and constant concern at the top. Scott D. Sagan, a political scientist at Stanford University, thinks he knows why such vigilance is hard to pull off. In the early 1990s, he set out to explain why the nation's nuclear-weapons program has been nearly accident-free. After unearthing classified documents and interviewing retired military personnel, however, he assembled a long catalogue of near-disastrous mishaps for his book The Limits of Safety: Organizations, Accidents, and Nuclear Weapons (Princeton, 1993). Mr. Sagan concluded that organizations don't always know after the fact why an accident occurred. He also found organizations sometimes treat mistakes as "political events for which credit and blame must be assigned." The mysterious explosion aboard the U.S.S. Iowa, which killed 47 sailors in 1989, was for two years officially blamed on a suspected saboteur and therefore prompted no safety reforms. Later, the Navy admitted that the explosion could have resulted from unsafe handling of weapons. Moreover, the people who make the mistakes can't be trusted to admit them, he says. And organizations tend to resist sharing sensitive and embarrassing information -- even internally. The theoretical dispute over whether we can make risky technologies virtually accident-proof may have reached an impasse. Optimists and pessimists can each point to particular industries or systems to make their case. But resolving that question requires far more empirical data than past case studies provide. The next wave of work in the field may be in the hands of number crunchers. Normal-accidents theory "is an extremely compelling way of looking at risk," says Andrew W. Lo, a professor of finance at the Massachusetts Institute of Technology's Sloan School of Management. "But Perrow's book is almost entirely non-quantitative. Whether or not organizations are subject to normal accidents is not a qualitative yes or no. It's a question of degree." Mr. Lo thinks that "interactive complexity" and "tight coupling" may be useful for explaining and avoiding global financial crises. Incidents like the collapse of Barings Bank in 1995 and last year's $2-billion hedge-fund bailout, he says, should be a wake-up call for our vulnerability to financial chain reactions. He is trying to interest Mr. Perrow in collaborating on ways to quantify his concepts. If only financial institutions and the Federal Reserve Bank would collect and publish the right sort of data, Mr. Lo says, he believes he could create mathematical formulas for measuring financial systems' exposure to risk. In the postscript for the new edition of Normal Accidents, Mr. Perrow does not soften the alarm he sounded 15 years ago, but he does see a progress of sorts. "Today it is almost routine for media stories to mention that there is no one single cause of accidents," he writes, "that while operator errors are frequently involved they are hardly a sufficient explanation of the accidents, and that we are dealing with complex systems where a series of failures came together in a way no one anticipated." But complex, tightly coupled technologies continue to proliferate, each one obliging us to decide, he contends, to shut them down, radically redesign them, or "live and die with their risks." http://chronicle.com Section: Research & Publishing Page: A19 Easy-to-print version E-mail this article