The RISKS Digest
Volume 9 Issue 16

Wednesday, 23rd August 1989

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Please try the URL privacy information feature enabled by clicking the flashlight icon above. This will reveal two icons after each link the body of the digest. The shield takes you to a breakdown of Terms of Service for the site - however only a small number of sites are covered at the moment. The flashlight take you to an analysis of the various trackers etc. that the linked site delivers. Please let the website maintainer know if you find this useful or not. As a RISKS reader, you will probably not be surprised by what is revealed…


Marc Rotenberg
Hazards of Airliner Computerization
Brinton Cooper
Risks, and an assumed definition of "reliability"
Bob Estell
Computers in Medicine
Brinton Cooper
Constructive criticism? Technology doesn't have to be bad
Donald A Norman
Tandem computers and stock exchange failure
Ernest H. Robl
TSE shutdown — a success story
Rich D'Ippolito
Incompatible IR controllers damage circuits?
David A Willcox
Re: a balancing act for wheel watchers
J. Eric Townsend
Keith D Gregory
Info on RISKS (comp.risks)


Tue, 22 Aug 89 16:53:25 -0700
The New York Times, August 12, 1989

Automated Planes Raising Concerns, by Carl Levin

Airlines are starting to fly a new generation of highly automated jets, raising
concerns among safety researchers that pilots will rely too much on the
technology and will lose or never learn the sharp skills and reflexes needed in

The first scientific study to compare pilots' performance in highly automated
and traditional cockpits began Tuesday In Atlanta. Researchers at the Federal
Aviation Administration and the National Aeronautics and Space Administration
said the results would help them improve training for pilots who fly the
advanced planes and suggest ways to better design future craft.

The most advanced planes, like the Airbus A320 and Boeing's 757 and 767 models,
require little of the hands-on flying skill that older models need. For years,
planes have had autopilots to keep level and make simple turns, but with the
newest equipment pilots can push a few buttons and lean back while the plane
flies to its destination and Iands on a predetermined runway. Virtually every
calculation is made by computer.

More Control to Machines

"We're taking more and more of those functions out of human control and giving
them to the machines," said Dr. Clay Foushee, the chief F.A.A.  scientist for
human performance Issues. "The question becomes whether humans will really
respond when something goes wrong."

Aviation experts cite the performance of the pilots of two disabled United
Airlines jets in recent months as examples of how basic flying skills and years
of experience can make a big difference in emergencies.

After a disintegrating tail engine crippled the hydraulic system on a DC-10 on
July 19, Capt. Altred C. Haynes, a 33-year United veteran, and his crew devised
a way to crash-land the plane in Sioux City, lowa. Of the 296 people aboard,
185 survived.

Capt. David M. Cronin, also a United pilot for three decades, cited his crew's
extensive experience in the safe landing of his Boeing 747 in Honolulu in
February after a cargo door and large section of fuselage blew off the plane,
knocking out two of its four engines and killing nine passengers:

Airlines See New Planes as Safer

"Here's two examples of unforeseen Qand in fact some engineers would have said
impossibleQtypes of failures that were dealt with creatively by human
operators," said Bob Buley, a flight standards manager at Northwest Airlines.
"If we have human operators subordinated to technology then we're going to lose
that creativity. I don't have computers that will do that; I just don't."

Airlines like the equipment because it keeps a plane closer to its course than
a pilot can, cutting costs and increasing safety in some operations.

The head of pilot training for United William H. Traub, said that the carrier
had been flying highly automated Boeing 767's since 1982 and that he knew of no
deviations greater than 300 feet from assigned altitudes. "In that respect it's
a safer system," he said in a telephone interview.

Leading the parade of new technology is the A320, a jetliner made by the
European consortium Airbus Industrie, which began passenger service in this
country last month. Northwest is buying 100 of the jets and has started flying
the first two. Braniff, with 50 on order and 50 on option, plans to put its
first A320 into service this month.

In addition to the increased use of automatic cockpit controls, the A320
eliminates virtually all direct mechanicaI or hydraulic Iinks to movable
surfaces on the wings and tail that direct a plane's speed and angle of flight.
Five computers translate a pilot's actions into electronic commands that move
surfaces, changing the plane's speed and direction.

Computers can control the speed and direction of flight more accurately than
any human pilot but even aviators who defend the A32O say the extensive use of
automation raises questions about a pilot's ability to respond quickly in an

Looking for 'Ideal Balance'

"In my perfect world we marry the advantages or automation and the creative
attributes or human operators," said Mr. Buley of Northwest.  "The A320 is a
quantum leap ahead.  What I'm looking for is the ideal balance, and I'm not
sure we've reached that with that airplane."

The equipment on the new planes is more reliable than before, and it can
relieve pilots of routine duties that might distract them from more important
tasks. For example, in the Boeing 767, computers automatically calculate and
adjust the descent speed to use the least fuel for the distance traveled, one
pilot noted. In the older Boeing 727, pilots go through "constant mental
gymnastics" to make the calculation themselves, the pilot said.

But just as some educators argue that a pupil with a calculator might not learn
basic principles of mathematics, aviation researchers say pilots who depend too
much on computers might not be as quick to determine the correct descent speed
on his own if the computer fails as would the 727 pilot who does this on every

Cockpit crew complacency and boredom are another issue, and these problems are
highlighted by a separate airline industry study of automation and pilot
performance. The F.A.A. and the National Aeronautics and Space Administration
are building on the airline study and the Atlanta research to develop a
national program to improve the ways technology is used in aviation.

Other concerns listed by the airline group, Ied by Mr. Buley of Northwest,
include the problems pilots face when automated equipment fails and the
deterioration of basic flying skills.

Pilots Share Concern

Pilots themselves share these concerns, according to a recent space agency
study of 200 pilots who have been flying the Boeing 757 for airlines.  about
half agreed with the statement "I am concerned about a possible loss of my
fIying skills with too much automation." Even so, nearly 90 percent of the
pilots agreed that the new instruments were "a big step forward."

Many of these questions will come up again Sept. 18-19 when Dr. Foushee and
representatives from manufacturers and airlines meet to discuss the national
plan for improving the way people use technology in aviation.

Adding to the urgency of the research is the current boom in pilot hiring. Over
the next decade a new generation of pilots will be climbing into cockpits, and
virtually all their airline training will come in the new jets.

"What happens when the automation fails?" asked Earl L. Wiener of the
University of Miami, an expert in pilot performance who is directing the
Atlanta study. "A collision is coming between very inexperienced pilots and
very sophisticated aircraft."

To be sure, today's pilots have the advantage of extensive training on advanced
cockpit simulators, which duplicate every movement a plane would make. A pilot
in a simulator can practice fIying after losing various computer and control

"There have been many simulator advances that hopefully will give pilots
training advantages that an older generation of pilots didn't have," Dr.
Foushee said.

Still, while simulator training could help for some kinds of emergencies,
others, like the loss of the hydraulic system in the DC-10 in lowa, are
considered so remote that pilots do not train for them on simulators.

Besides examining how well pilots respond in emergencies, researchers hope to
examine any differences in the ways pilots work with one another in automated
and conventional cockpits, said Dr. Everett Palmer of NASA's Ames Research
Center, which is financing the study led by Dr. Wiener.

Hazards of Airliner Computerization

Brinton Cooper <abc@BRL.MIL>
Tue, 22 Aug 89 17:04:57 EDT
Mike Trout quotes BBC News, ...  pointing out that flight crews need to do
something "critical" to the success of the flight.  The solution may be right
under our noses.

How often, in this forum, have we discussed applying tests of "reasonableness"
to computer-generated answers to problems?  It seems that such tests are
critically needed in the cockpit, the most obvious example being Flight KAL
007.  Such reasonableness checks as humans would be capable of performing,
would be far from "make work" and would reduce significantly some of the risks
associated with increasingly automated flying.

...or so it seems from here.

Risks, and an assumed definition of "reliability"

22 Aug 89 15:51:00 PDT
RISKS 9.15 highlighted a phenomenon [I am tempted to say, "problem."]
that I've noted in RISKS for some time:
 We tend to want computerized systems to be very much more reliable than
 non-computerized systems.

For example, is my Seiko watch reliable? Yes.  Has it ever failed?
Yes; the original battery ran down after 5 years.  Does it run
exactly in synchronization with the Naval Observatory master clock?
No; it gains about a second a week.  Is that OK?  Yes!  It's great!

Is my old '66 Pontiac reliable?  Yes.  Has it ever failed? Sure; batteries
have gone dead; a tire blew out; water pump failed (at about 90K miles);
alternator failed (about the same time); tune ups needed every 3 years;
... I've probably belabored the point too long already.

Folks, we've been spoiled by our own successes.  I'm all upset with the
maker of the hard disk in my Mac II, because it needs to be replaced after
only 18 months.  I have to stop and think about where I began, in 1960.
The computer we had then was less than 10% the horsepower of my Mac; it
had a miserable collection of "user tools" - the best being a FORTRAN II
(yes, "2") compiler.  And it went down at least 4 hours every week for
"maintenance."  And it cost a million dollars (or so); so the whole base
[not China Lake] got by on just one.

But can we improve?  You bet.  My Norelco shaver [first one] lasted 7 years;
that's a lot better than the hard drive in either my old Mac +, or this Mac II.
Maybe Norelco should teach "brand X" disk drive maker about motors?

Computers in Medicine

Brinton Cooper <abc@BRL.MIL>
Tue, 22 Aug 89 23:56:50 EDT
We seem to have more than our share, in the Digest, of horror stories
about computer failures in stock exchanges, motor vehicle records,
aircraft control systems, weapons control systems, and banking
applications.  While I have not sampled the subjects scientifically, it
seems as if we've not had quite so many horror stories in medical
applications.  (Of course, I'm not asserting that there have been none!)

Several years ago, a few colleagues from the Lab and myself consulted
with the Shock-Trauma Unit of the Maryland Institute for Emergency
Medical Service Systems regarding their use of computing in clinical
applications.  At the time, they used a DEC computer for patient
records, testing results, pharmacy and medical orders, etc.  It included
software which "integrated" these applications so that the attending
physician could trace the effectiveness of medicines and therapies with
time.  (Shock-Trauma gets the most seriously injured patients, often
flown in by the MD State Police in helicopters landed on premises.)

An important concern of the medical personnnel at the time was computer
failure.  Although the physicians initially resisted the machines'
intrusion into their domain, they had ultimately been "won over" and
became quite fond of and dependent upon the computer.   Virtually every
medical person in the unit learned, voluntarily, how to re-boot the
system in the event of a crash — which was relatively often.  This was
the late 1970s.  They wanted to know if they could justify the funds to
purchase a fully redundant system.  There were two interchangeable
computers doing different functions (one was not critical).  They felt
that a third machine would give them the security of always having their
data available, but they needed justification and support from so-called
"experts," i.e. us.

Well, we gave them what they needed, but that is not the point of the
story.  The points are:

    1.  Is my perception correct?  Are there proportionally more
life and property threatening computer-related faults in banking,
transportation, and national defense than in medical applcations?

    2. If there's even a modicum of truth in #1, then why? Certainly
the hardware and software aren't unique in the hospital.   Is it a
matter of how they're used?  Is there more emphasis on redundancy and
reliablilty and less on moving it faster and making another buck?  Are
the machines introduced into new applications more gradually, so that
users are assured of correct operation at every step of the way?

    3. Or are the physicians merely burying their mistakes again?


Constructive criticism? Technology doesn't have to be bad

Donald A Norman-UCSD Cog Sci Dept <>
Wed, 23 Aug 89 09:47:53 PDT
I like the Petroski book ["To Engineer is Human: The Role of Failure in
Successful Design"].  It is an excellent example of design and the problems
that are inherent in pushing technology beyond what science can yet (ever?)

This is especially true when people chide me about human interface technology
and say something like "How come interface design isn't `scientific', like, say,
bridge design."  I tell them to read Petroski and then tell me about bridges.

I recommend Petroski to all my friends and students.  (I am happy to say that
someone told me that he, in turn, recommended my book.)

Tandem computers and stock exchange failure

Ernest H. Robl <>
Wed, 23 Aug 89 11:49:57 EDT
The quoted reports on the problems with the Tandem system at the
Toronto Stock Exchange are a good example of the difficulty the
news media have with reporting on complext technological stories.

As someone who works with a Tandem system, I can point out a few
things that may be of value to Risks readers:

Tandem computers do not have "backup systems" as such.  Instead,
the design incorporates redundant components — all of which
perform work under normal conditions.  The minimum system Tandem
will normally sell you is one with TWO CPUs, and at least one of
the discs ($System — the one with the operating system) mirrored.

How failsafe a system is depends a lot on how the system is
configured.  Most Tandem systems have at least some of the discs
unmirrored.  (That's usually an economic decision.)

With mirrored discs, data is always written to both discs.  However
since it neads to be read from only one, there are situations
where different reads can be performed at the same time on the two
halves of the mirrored pair — which will actually provide a gain
in performance for some operations.

Based on the quoted reports, I assume that the failures at the
stock exchange involved both halves of a mirrored disc pair --
though that's not obvious.  I'd be interested in hearing additional
details, if they are reported.  (Mail to me, if you don't think
this is of interest to the Risks audience.)

My opinions are my own and probably not IBM-compatible.--ehr
Ernest H. Robl  (ehr@ecsvax)  (919) 684-6269 w; (919) 286-3845 h
Systems Specialist (Tandem System Manager), Library Systems,
027 Perkins Library, Duke University, Durham, NC  27706  U.S.A.

TSE shutdown — a success story

Wed, 23 Aug 89 12:38:08 EDT
In RISKS 9.15, Peter Roosen-Runge brings us the following quotes:

 A computer crash all but shut down trading on the Toronto Stock Exchange for
 almost three hours yesterday, forcing tens of millions of dollars' worth of
 trades to Montreal. ... [the crash] — a multiple failure within a disc-drive
 subsystem — forced a halt at 9:41 AM. ... `Two pieces of hardware break down
 and Bay Street breaks down.' said a sour Charles Mitchell, trader. ... `Who's
 accountable for that?' ...

 A TSE spokeswoman said the failure of both primary and backup systems had
 never occurred since computers were installed 26 years ago. ...

My rough calculations indicate that the system availability has been 99.9986%
for those 26 years.  Who, indeed, IS responsible for that — give them a

 `Everybody's very much annoyed,' McLean McCarthy's Mitchell said. `It's
 costing us a lot of money. I think the people upstairs in the exchange should
 be held accountable for it.'

How much were these people making with the chalkboard system?

It is human nature to demand perfection from everyone and everything else.
Have these folks ever heard of business insurance?  It should have been very
inexpensive given the prior availability of the system.

Along with our efforts to reduce risks in our trade, do we not need to
educate users in risk management?

Rich D'Ippolito

Incompatible IR controllers damage circuits?

David A Willcox <>
Wed, 23 Aug 89 09:36:15 -0500
I few weeks ago, I spent a couple of nights at a fairly nice hotel on
the East Coast.  You could tell it was a nice place because the remote
control for the TV was not bolted to a table.

I was intrigued by the notice that was pasted to the remote:

    CAUTION: The frequency of this television remote will
    damage the internal electronics of any set not programmed
    to receive the spectradyne signal.

My first reaction was to chuckle at this rather obvious attempt to
scare light-fingered but gullible clientele out of "offing" the remote.
But I got to wondering.  Is there any possible truth to this?  If
there is, how do I know that my VCR remote, say, won't damage my TV?
And if my TV was damaged, wouldn't that be evidence of really poor design?

I suspect that the worst "risk" here is that some guests of this hotel
are going to get a very warped idea of reality.

Re: (a balancing act for wheel watchers)

J. Eric Townsend <>
21 Aug 89 17:49:14 CDT (Mon)
Actually, many computerized balancing/alignment systems are very, very simple
(even for mechanics :-).  Monitoring devices are attached to the wheels while
the car is on a lift.  The car's data is looked up in a book and entered in by
hand on a large number of the machines.  (I have a car not in "the book" and
have had to provide my own data.)  Then you procede to align/balance by looking
at a rather basic "under/correct/over" meter for each wheel.

There are probably a half-dozen other ways to do balancing/alignment, and
probably a thousand variations on the above theme...

J. Eric Townsend, 511 Parker #2, Houston, Tx 77007

Re: Tired of computers being trusted? (a balancing act for wheel watchers)

Keith D Gregory <>
Tue, 22 Aug 89 09:19:54 edt
More likely, the mechanic was not "machine literate".  I ran into a similar
problem: I had a flat repaired, and the shop (run by the company that made the
tires) balanced the tire as part of the repair.  At the same time, I purchased
a "lifetime balancing and flat repair" contract.  Driving home, I noticed a
slight shimmy that wasn't there that morning.  The next morning, I took the car
in for a complete rebalancing.

And when I drove home, the shimmy was worse - much worse.  The next day, I went
back to the shop and complained.  This time I watched as the tires were
balanced.  What had happened was that the "mechanic" (I use that term loosely)
did not have a properly sized chuck for the wheel balancing machine.  So he
used one that was "close".  As a result, the wheel was able to move from side
to side while it was being tested, with the result that the weights were put in
random (?) locations.

The moral?  If you don't trust computers, don't trust the people that do.
    [So it would be very easy to key in wrong data for the given car, or
    correct data for the wrong car, etc.  Thanks.  PGN]

Please report problems with the web pages to the maintainer