The Risks Digest

The RISKS Digest

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 9 Issue 20

Friday 1 September 1989


o Last Night's "Tonight" was Unknighted; Cars on Carson Top Hat Trick
o More on the Therac 25 -- by Jon Jacky
o Witness questions attack on Iranian jet
Robert Dorsett
o Risks of on-line course registration
Deborah M. Clawson
o Specifications
Martyn Thomas
o Re: Lowest-bidder or weak specs?
Robert Hirsch
Bill Cattey
o Pilot simulator training and boredom
Dan Franklin
o More on automation
Robert Dorsett
o Info on RISKS (comp.risks)

Last Night's "Tonight" was Unknighted; Cars on Carson Top Hat Trick

Peter Neumann <>
Fri, 1 Sep 1989 11:00:36 PDT
On 31 August Johnny Carson did an impression of Abe Lincoln as a budding young
standup comic, wearing a two-foot top-hat.  The end of the skit involved
triggering a radio-controlled device that blew the hat upwards off of his head.
A long delay arose during the taping of the show when the hat kept blowing off
Johnny's head backstage before his entrance.  His technicians figured out that
RF interference from nearby kids with remote-control robot cars kept triggering
the hat.  The obvious solution was to get the kids to stop transmitting during
the skit.  Honest, Abe had some good lines, but Johnny's hair-raising gimmick
top-hatted them all.

More on the Therac 25 -- by Jon Jacky

Peter Neumann <>
Fri, 1 Sep 1989 11:06:32 PDT
There is a very nice article by Jonathan Jacky entitled "Programmed for
Disaster, Software Errors That Imperial Lives", in THE SCIENCES, the bimonthly
of The New York Academy of Sciences, September/October 1989.  Jon gives an
in-depth analysis of the Therac 25 case.  (The credit line notes that Jon is
currently designing softwre for a computer-controlled radiation-therapy machine.)

Witness questions attack on Iranian jet

Robert Dorsett <rdd@rascal.ics.UTEXAS.EDU>
Fri, 1 Sep 89 12:00:00 CDT
By George C. Wilson
Washington Post Service

The USS Vincennes, which shot down a civilian airliner in the Perian Gulf on
July 3, 1988, killing 290 people, had gained a reputation for being an overly
aggressive "robo-cuiser" and probably provoked the sea battle with Iranian gun-
boats which preceded the shootdown, according to a commander of a ship on the
scene at the time.

Writing in the September issue of the US Naval Institute magazine _Proceedings_,
Commander David Carlson said the radar on his ship, the USS Sides, a patrol and
escort frigate, showed that the airliner was climbing in a non-threatening way
inside the designated commercial airline corridor.

He said he would not have ordered his own crew to fire missiles at the plane,
as did Captain Will Rogers III, skipper of the Vincennes.

In another part of his remarkably frank account, which is likely to refuel con-
troversy generated by the shootdown, Carlson wrote that the Vincennes started
the sea battle in response to what may have been nothing more than warning
shots from Iranian gunboats feeling threatened by the cruiser's helicopter.

Having watched the performance of the Vincennes for a month before the
incident," Carlson wrote, "my impression was clearly that an atmosphere of
restraint was not her long suit.  Her actions appeared to be consistently
aggressive and had become a topic of wardroom conversation.  "Who's driving
the problem in Vincennes?" was a question asked on numerous occasions prior
to 3 July.

"'Robo-cruiser' was the unamusing nickname that someone jokingly came up with
her, and it stuck,' Carlson said.  My guess was that the crew of the Vincennes
felt a need to prove the viability of Aegis (the highly sophisticated anti-
aircraft system on the cruiser) in the Persian Gulf, and that they hankered
for an opportunity to show their stuff..."

The Vincennes crew misidentified the civilian airliner as an Iranian F-14
fighter plane, which does not carry anti-ship weapons, and shot it down with
missiles, to the horror of Carlson watching the same plane on radar.

"We locked up and illuminated" the approaching Iranian airliner "with our
missile fire control radar," Carlson wrote.  "The aircraft continued climbing
on a southwesterly course that would take it directly over the Vincennes
position.  Based on closest point of approach to the Sides (range and
altitude), lack of any significant known F-14 antisurface warfare capability...
lack of detected radar emissions and precedent, I evaluated the track as a non-

"The Vincennes announced her intentions to take TN 4131 (Track NUmber 4131, the
Iranian airliner) with missiles at 20 miles.  I wondered aloud in disbelief,
but I did not do the one thing that might have helped.  I did not think to push
for a re-evaluation of IFF (identification of friend or foe)..."

Risks of on-line course registration

"Deborah M. Clawson" <Clawson@DOCKMASTER.NCSC.MIL>
Fri, 1 Sep 89 17:01 EDT
Yesterday the University of Colorado at Boulder newspaper carried an article
entitled "Officials say CU keeps records confidential." The article was in
reaction to reports that a murderer obtained his former girlfriend's course
schedule by giving a University of Washington clerk the victim's Social
Security number and name.  The point of the article was to assure students that
"that couldn't happen here." In interviews CU officials reviewed employee
training on the Family Education Rights and Privacy Law and said that reminders
about the law are consistentlly sent out.  A course schedule would not be
released to anyone other than the student and only if the student had ID.
Every one of administrators interviewed was confident that no one else could
obtain a student's course schedule.

The RISK involved is that no one seems to have thought about our on-line
course registration system.  This system allows a student to review his
or her schedule by phone.  And what do you need to access that
information?  The student's Social Security number and a secret PIN.

Unfortunately, the PIN is assigned as the student's birthdate...


Martyn Thomas <mct@praxis.UUCP>
Thu, 31 Aug 89 18:58:29 BST
>From: David A Honig <honig@BONNIE.ICS.UCI.EDU> [writing in RISKS]
>It seems to me that if the specifications are complete and correct then
>there should be no problem.

It seems to me that much of the grief in our industry, and many of the
risks, stem from the assumption that every system has an "a priori"
specification, if only you can find the right person to ask.  In my
experience, specifications change as people understand the implications of
what they have asked for. Specifications also change because new ideas come
along, or the users change, or the constraints change.

Most of all, specifications change because every useful system mirrors the
real world in some aspect, and the real world changes. So specifications (in
the sense of *real, current needs* ) change throughout the development of any
useful system and throughout its service life, too.

This is why "lowest compliant, fixed-price bid" is a poor way of purchasing

Re: Lowest-bidder or weak specs? (David A Honig, RISKS-9.18)

Tue, 29 Aug 89 09:33:08 EDT
How about "most trusted bidder with a reasonable price"?  The problem
with many lowest bidder schemes (particularly those often legislated by
well-meaning government bodies) is that they do not provide an adequate
mechanism to separate competent bidders from incompetent bidders.  What
good are detailed specifications if you don't honestly believe that the
contractor will be able to meet them, or if you believe you're going to
have to pay enormous legal fees to force the contractor to meet them?
There's plenty of opportunity for abuse if you let public officials
hire their "favorite" contractor regardless of price, but a pure lowest
bidder scheme is not the solution, either.

Re: Lowest Bidders (RISKS-9.18)

Robert Hirsch- RCE <>
Tue, 29 Aug 89 09:54:24 PDT
A somewhat jaundiced view of the process from the viewpoint of an employee of
several large and small contractors (who shall remain nameless) is that first
of all, the going practice seems to be to ask the developers what it will cost
to develop a system to the spec provided, then say "no, try again" to realistic
estimates until the bid comes down to less than the program office believes the
competition will bid.  The same, of course, applies to schedule.  In short,
it's a low-ball contest with the award going to whoever makes the most
audacious bid.

Next comes the engineering challenge of devising an implementation which
can be done in the time and budget which was bid, and which can somehow
be construed as meeting the requirement.  A specification calling for
on-line error diagnostics may be met by having the terminal beep, or
some such travesty.

Of course, there is the problem of divining what the spec does ask for.
I have been at a design review where our customer was a procurement
agency who was acquiring a system on behalf of the end-user agency, and
there were anguished cries from the end users who didn't even recognize
the system the bean counters had specified, let alone what we were
proposing to provide.

The solution to all the above comes about in engineering change
proposals in which the necessary features are finally added at a price
which brings the total cost about up to where it should have been bid in
the first place.

Bob Hirsch, Sterling Software, NASA Ames Research Center,
Mail Stop 233-10, Moffett Field, CA 94035  415 694-4807

Re: Lowest-bidder or weak specs? (RISKS-9.18)

Bill Cattey <>
Tue, 29 Aug 89 14:47:58 -0400 (EDT)
I think David Honig has pointed out a kind of naivety about the process of
quality software that allows government and management to persist procurement
through lowest bidder.

> It seems to me that if the specifications are complete and correct then
> there should be no problem.

The problem is that specifications are never complete.  The human users
and implementors are always finding changes that need to be made.  The
procurement process must take into account that the spec will evolve
over time.  The procurement is not only for the product specified, but
for the team that builds it, and revises it as it goes along.

> And I cannot think of another decent way to buy systems (the
> next-to-lowest bidder?  the highest bidder?)  that will solve the

Indeed, this is a hard problem.  Here's a suggestion for a procurement
process that will do a little more to keep vendors honest:

1. Sealed Bids (same as now)

2. Discard the highest and lowest bids. (Since all bidders are bidding
on the same job, if the bids are wildly out of range there's either a
problem with the spec or the vendor.)

3. Have those who produced the spec visit the remaining vendors and
judge whether the vendor can do what they say they can.

4. Have management choose either the lowest bidder, or the highest
quality based on the recommendation made in the previous step.

Any procurement process that reduces to a simple mechanical process engenders
RISKS.  Although there is the possibility for human error in the process I have
suggested, it is designed to address the faults of the existing procurement
process of "lowest bidder", while minimizing the risk of doing worse than
"lowest bidder".

Pilot simulator training and boredom

Dan Franklin <dan@BBN.COM>
Wed, 30 Aug 89 22:26:29 EDT
If, on the one hand, pilots in modern automated planes risk falling
asleep from boredom, and on the other hand, they never seem to get
enough simulator training in really oddball failures and problems,
maybe future cockpits should be designed with simulation software
built in, to allow them to practice while they fly.

They couldn't be as good, naturally.  Real cockpit simulators simulate
everything, including the motions of the simulated airplane, but perhaps going
a few rounds with something more modest, along the lines of a video game, would
be useful nonetheless.  And it would certainly keep the pilot awake!  You just
need to make sure the simulation is never confused with the real thing...

    Dan Franklin

More on automation

Robert Dorsett <rdd@rascal.ics.UTEXAS.EDU>
Thu, 24 Aug 89 10:34:57 CDT
>However, the article does not objectively cite balancing situations in which
>automated systems handled sets of conditions that were too complex and changing
>too fast for humans ever to respond satisfactorily.

That's probably because such situations don't tend to exist.  Automation in
airliner cockpits serves to reduce workload, such as when entering congested
terminal areas.  To insinuate that the pilots *could never* handle such sit-
uations is to ignore that single-pilot general aviation IFR operations do it
all the time.  There may be two exceptions to this generalization:

(1) Extending routine automation into critical weather situations, namely
0/0 (Category III) landing.  Airliners can now land in weather that most of
us wouldn't be caught dead driving in.  Think about it...:-)

(2) A secondary layer, usually transparent to the pilot, when his airplane
itself is so aerodynamically unstable that he could not hope to control it in
a consistent manner over time.  In this case, the flight-control system "sim-
ulates," in-flight, either traditional (as in the case of the 747, which tries
to provide the "feel" of a 707, through control-force dampers or amplifiers)
or untraditional (A320/330/340) flying characteristics.

<>  Still, while simulator training could help for some kinds of emergencies,
>One wonders, "Why not?"  Can the great minds of the aircraft industry not
>postulate virtually any kind of emergency?

Writing simulators is extremely difficult.  The computational ability may
not be available to accomodate higher-fidelity simulations, even if a satis-
factory method existed to write the billions of lines of code required to rep-
resent *every* fault situation.  The most cost-effective method, and the current
trend, has been to identify those factors which are most likely to occur in
a flight (emergency and otherwise), and represent them as realistically as
possible.  In certain cases, "classic crashes" (such as windshear incidents)
may be simulated and pilots run through them.

Rolfe's text, _Flight Simulation_, is a nice overviews of the problems and
promise of flight simulators, both military and civil.

>For years, the US Air Force has been
>flying high performance tactical jets using significant amounts of automation.
>If the automated system fails, the game is over.

The automation is an essential component of the flight control system for these
types of aircraft.  And if it fails, like you say, the game is over: EJECT.
That option does not exist in civil situations.

>The entire article is a good example of the warped treatment afforded a
>complex, scientific topic by the popular press.

The article was largely on-base.  If anyone would like to see what other "so-
called experts" have to say, I recommend Nagel and Wiener's _Human Factors
in Aviation_ (Academic Press: 1988, $65 or so, hardcover).

Please report problems with the web pages to the maintainer