The Risks Digest

The RISKS Digest

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 4 Issue 62

Wednesday, 11 March 1987


o "Software Safety: What, Why, and How"
Minireview by Jim Horning
o Beef with Restaurant's Hi-Tech Computer
Yigal Arens
o Electronic Steering
Mike Brown
o Enhanced 911 risks
Mike Brown
o Computers in the arts
Don Craig
Glenn Trewitt
o Mode C
Ken Calvert
o Re: Plane Crashes
Ronald J Wanttaja
o Re: Results of a recent security review
Arnold D. Robbins
o Risks of Maintaining RISKS — and a reminder for BITNET readers
o Info on RISKS (comp.risks)

"Software Safety: What, Why, and How"

Jim Horning <horning@src.DEC.COM>
Wed, 11 Mar 87 19:41:04 PST
      Capsule Review of "Software Safety: What, Why, and How",
      Nancy G. Leveson, ACM COMPUTING SURVEYS, Vol. 18, No. 2

  "This survey attempts to explain why there is a problem, what the problem
  is, and what is known about how to solve it. Since this is a relatively
  new software research area, emphasis is placed on  delineating the
  outstanding issues and research topics." [From the Abstract]

I've often wished that there was a good survey of the RISKS field that I
could recommend to people who weren't as well informed on the subject as
I thought they ought to be.  I'm pleased to report that such a survey has
now been published in a widely accessible journal by a frequent RISKS
contributor.  Software safety is the central theme, but many other aspects
of risks to the public in computer systems are mentioned.  ("Mishaps are
almost always caused by multiple factors, and the relative contribution of
each factor is usually not clear.")

Regular readers of RISKS will find much of the material familiar.  But they
should appreciate the careful documentation of numerous software-related
mishaps and the perspective obtained by pulling back a little from our daily
batch of anecdotes and looking for general issues and principles.  The paper
is well organized, and lucidly written.  More than 100 references point
interested readers to more detailed and/or authoritative information.  There
is also a general bibliography with 28 entries.

Despite the title, this paper doesn't tell how to produce absolutely safe
software.  Instead, it tells how hard it is, and why it is hard. Activist
readers of RISKS will probably feel that it stops just short of some
obvious conclusions and calls for needed action.  (However, I'm sure that
it would not be possible to get a consensus of this group about what the
"obvious" conclusions and actions are.)

(This issue is dated June 1986, but my copy arrived last week. That this
is a Publishing Risk, rather than a Postal Risk, is indicated by an
apology printed at the front of the issue.)
                                                  Jim H.

       [Nancy herself proposed a solution to the last-mentioned problem --
       simply declare the June 1986 issue to be a special combined issue
       dated June/ September/ December 1986/ March 1987.  However, 
       subscribers might feel cheated.  PGN]

Beef with Restaurant's Hi-Tech Computer

Wed, 11 Mar 87 12:58:17 pst
This happened to me several months ago, but I only just realized that it
might be of interest to this group.  My wife and daughter and I went to a
rather fancy restaurant here in LA.  We all ordered steak-type dishes and
specified various degrees of doneness, all on the rare side.  As we were
eating our first courses there was a short blackout, lasting approximately two
minutes.  The restaurant remained illuminated by the candles on the tables.

When the main courses came they were very overdone.  Our waiter took a
peek at the food and then called the manager.  The manager apologized
profusely and blamed the computer that controlled the kitchen(!).  As
far as I could figure out, he was claiming that the blackout wiped out
the memory of a computer which (among other things?) controls cooking
times.  When the power returned, the chefs had to try and recall how
long things had been cooking, and some mistakes occurred.

I have no idea if this was the truth or whether the manager simply
thought that a high-tech excuse would be most effective.

Since we were too hungry to wait for new dishes, we ate what we had
received anyway.  We were not charged for the meal.

Yigal Arens

   [This does indeed suggest a completely new line of high-tech excuses.  PGN]

Electronic Steering

Wed, 11 Mar 87 09:27:29 est
I haven't had the opportunity to follow all of the discussion on the
electronic steering issues that have been in the Risks Forum.  I notice that
there seem to be two sides: those who are scared and those who believe that
the manufacturers will develop a safe product.  Look back at several of
those things that manufacturers should have caught: Pinto gas tanks, Audi
5000's sudden acceleration, Ford E-350 ambulances and their propensity to
catch on fire.  There are thousands of examples of such potentially
catastrophic hazards that have made it through the design and development
into manufacturing.  Every car manufacturer has had problems of this nature.
Yet, the tools and techniques for safety analysis of hardware have been
around for many years.  They can be very effective if properly applied.  Yet
we still have problems cropping up.  Now we are proposing to allow steering
to be controlled by software, control systems for which we do not have the
tools and techniques that exist for hardware systems.  Every software
engineer will tell you that it is impossible to eliminate all of the defects
in software: therefore, we have to ensure that the defects that remain do
not cause a safety problem.  Not a simple task.  The process has to start at
the concept stage.  The software requirements must take into consideration
the failure modes that can occur and develop traps to ensure that the system
fails safe.  The implementation of the safety requirements in the software
requirements must be thoroughly analyzed and tested.  Even then, it is
difficult to develop a "warm, fuzzy feeling" about this system.  Years of
development can be destroyed by a simple failure that results in a fatal

Mike Brown, Chairman, Triservice Software Systems, Safety Working Group

Enhanced 911 risks

Wed, 11 Mar 87 09:13:27 est
Several people have commented recently about errors in enhanced 911 systems
that resulted in misdirecting police, fire or rescue personnel.  In these
instances, a big safety issue is present.  In my capacity as Emergency
Services Coordinator for my county, I have been involved in investigating an
enhanced 911 system for us.  The system that has been proposed offers the
dispatchers the capability of entering information in addition to the
physical location of a caller into a master database.  This database is
located quite some distance from our locale and services a number of
jurisdictions.  Suggestions that have been made by our fire and rescue
personnel include inserting information such as handicaps or special medical
problems of residents, special problems that may be encountered in gaining
access to people (e.g., having to ford a stream), etc.  As the "good ideas"
expanded, it was suggested that people who may have toxic materials (farmers
with farm chemicals) or other hazardous materials (we have a number of gun
clubs that store black powder or reload their own ammunition), gun
collectors or others who may have valuables in their home, etc. be included
in this database.  Immediately I had visions of someone misusing the
database to commit crimes, etc.  How do we ensure the security of a database
of this nature when the people who are required to have access to it cannot
be trusted?  Recently, two local jurisdictions have had sheriff's deputies
arrested for participating in a burglary ring that has been functioning in
the area for 15 years.  Scary, isn't it?  And then there's Big Brother....

                Mike Brown

      [Maybe that is the same gang that was rampant in New Jersey in the
      1960s, giving free estimates on police-linked burglar alarm systems,
      after detailed on-premise inspections , with the more profitable 
      houses being burgled if they did NOT subscribe.  PGN]

Computers in the arts [Manual vs computerized lighting systems]

Wed, 11 Mar 87 02:07:37 PST
In my youth I worked as a stage manager and lighting designer/operator
for a number of summer stock (and winter broth) companies.  The most
complex show we ever pulled off had about 300 cues over a two hour
period.  (I do think the art suffered as a result.)   This was on
pre-computer but modern (1966) equipment, with 72 x 6 KiloWatt electronic
dimmer channels, and 4 presets (four slider pots per channel).  The
control room contained a desk with master controls, and a side-wing with
288 slider pots on it.  On our 300 cue show, we had 3 people operating...

When I later worked for the Canadian Broadcasting Corporation in Montreal
(1972), the lighting system for Studio 42, a 600 seat auditorium, had 600
12 KiloWatt channels (one per seat :-).  The channels were connected via a
48 volt patch panel to 80 control levels.   The patch panel was a 600 by
80 matrix wherein the operator inserted a pin to make a connection.
(Multiple assignments picked the lowest numbered control level).
The 80 control levels had a primitive computer system for storage/recall,
but the one slider pot on the control desk could be driven to the level
of a recalled channel by a motor, and would detect the operator's touch
(capacitively) to permit the setting of levels.

Neither of these systems ever had all channels working at the same time.
The electronic dimmers were packaged in racks and racks of drawers, and
it was a simple matter to repair a vital channel by moving a drawer.
The 1966 vintage Strand Electric console had sufficient internal parallelism
that we lost channels and pots, but never the whole thing.  (I vaguely
recall a dual power supply).   The CBC system was another story, and
a fully qualified Group 8 (the top pay scale) NABET technician was always
in the building when Studio 42 was in use.  The lighting system failed
occasionally, but never took more than an hour or so to restore to operation,
since we stocked a full board replacement inventory.  In my five years there
it never failed on air.

I see current lighting control systems at the television trade shows.
They use multiplexed twisted pair to connect a small control desk
to `intelligent' dimmer boards.  Smaller systems build the dimmers
right in.  The control desk contains a microprocessor that operates
each channel, and reads levels from a floppy disc.  The key to making
these systems redundant is buying two control desks.   Individual
dimmer channels can fail, but that won't shut down a show.

The amount of rehearsal needed to choreograph the operation of a manual
lighting console is significant.  A failure of the modern control desks means 
the system is down, since there aren't manual controls any more.  (It's
usually possible to wire a channel on.)  The technical solution is simple (buy 
or rent another control desk for performances) but the people making these
decisions are often not technical (trust me), and view such backup as a luxury.

Don Craig, Tektronix Television Systems

Computers in the arts — The Show Must Go On.

Glenn Trewitt <>
11 Mar 1987 1113-PST (Wednesday)
One of the most painful memories that I have:

Last fall, I attended a Pilobolus modern dance performance at Berlekey.
Their last segment was a performance of "Carmina Burana", accompanied by a
compact disk.  This is a long piece, perhaps 20 minutes.  About two-thirds
of the way through, when the performers were dancing "blind" (they had
various things on their heads), the disk skipped to a different section.
Many people didn't notice this, but I had seen the performance before and
listened to the music at home.  The performers were REAL good — they
recovered perfectly and the show went on.
                                            [What an ORFFul experience!  PGN]

For about 2 minutes more, that is.  At that point, the disk went nuts and
started playing random 10-second bits of music.  Generally, just enough for
the performers to start to recover.  This went on for about a minute before
they gave up and turned off the "music".  But it seemed like an eternity,
with the poor dancers up there on stage, just thrashing.

I see the risks here as different from other risks associated with any other
pre-recorded music, because almost all other failures are not so
catastrophic.  With a record, for example, you can just nudge the needle and
continue.  On the flip side, it occurred to me that it was quite possible
that someone in the audience had, with them, the technology to fix the
problem.  Namely, a portable CD player, perhaps with the same CD.  An
amusing thought.
                                - Glenn Trewitt

    [Although off the subject of RISKS, this is of course a common
    failure mode of CDs — not just a skip and a jump, but wildly
    erratic behavior.  Similar things happen in digital computer control 
    systems (as opposed to analog) — slight errors may translate into
    wildly erratic behavior, e.g., a wild control transfer...  PGN]

Mode C

Ken Calvert <>
Wed, 11 Mar 87 09:41:53 CST
[Me on Karn on mode C for all:]  (RISKS 4.61)

  > ... I expect that most such planes virtually never enter the busy airspaces
  > (i.e., Terminal Control Areas) where midairs tend to occur.  One reason
  > is that regulations ALREADY require radios and transponders for aircraft
  > operating in TCAs, as well as permission from the controlling authority.
  >       [These last two sentences reach an apparently false conclusion.  
  >       (For example, Los Angeles and Chicago routinely report many such 
  >       incursions each day.)

I don't see how my conclusion is false.  My conclusion was NOT that
incursions do not occur.  The point is that an Airplane Without A
Transponder is not a greater threat to other aircraft IF it never goes in
airspace where a transponder does any good (busy terminal airspace or
airways).  Moreover, I have not seen anything to indicate that all or even
most incursions into TCAs are made by Airplanes Without Transponders.  Have

     [Absolutely.  There seems to be lots of evidence that the incursions
     are by dingbat pilots, generally without appropriate avionics (adequate,
     nondefective, ...).  In the Aeromexico case, the controller was totally
     distracted by dealing with one dingbat, and ignored another — BOTH of
     whom were transgressing.  PGN]

In my understanding (as a temporarily inactive Private Pilot) the only
thing that requiring Mode C on all aircraft does that the current
regulations don't is require Mode C on aircraft NOT operating in busy
airspace. If the current regulations don't work, this won't either.
As you noted, there's a difference between regulation and reality.
Transponders have to be turned on to work.  Clearly some TCA incursions
are made by planes with Mode C transponders - irresponsible/incompetent
pilots may fly all kinds of airplanes.

On the other hand, if the proposal is to have all aircraft equipped with
Mode C that operates AT ALL TIMES, then the proposal must also require
ATC to monitor all aircraft.  As others have noted, I think it
will be some time before the system can handle that, although that may
be a worthy goal.  Even then, incursions will occur.  And your
comment applies here:

>...rely too much (or blindly) on the technology, then the existence of the
>   technology may be debilitating.  PGN]

As a side note, my brother has been training to become an Air Traffic
Controller for about nine months.  He won't even sit down at a radar screen
for a long time yet.  New controllers must be completely familiar with the
"old" manual system, which is of course used when things break down
(actually it is always used; radar and computers are simply an aid).  My
impression from speaking with him is that the ATC system has a healthy
distrust of (at least some kinds of) technology.

Ken Calvert, Univ. of Texas Computer Sciences

Re: Plane Crashes (RISKS 4.56)

Ronald J Wanttaja <ucbcad!ames!ll-xn!ames!uw-beaver!ssc-vax!wanttaja@ucbvax.Berkeley.EDU>
Wed, 11 Mar 87 08:49:52 pst
> In Europe there was a spate of (F-111?) crashes.  The apparent cause of
> these crashes was pilots (1) believing they could fly the plane on their
> own without the help of any dumb computer, (2) turning the computer off,
> and (3) promptly flying into a mountain.
> Any Hints?                 DavidP

Don't know much about this particular case, but there is a famous story
about the early days of the F-111 in Southeast Asia...

The F-111 was the first fighter-bomber with Terrain Following Radar.  The
radar controlled the plane through the autopilot to "hug" the earth; flying
about 200 feet above ground level.  It would look far enough ahead, and if
an obstruction was sighted, the plane would pull up at the appropriate
moment at a pre-programed G level (for those interested in further details
of this type of flying, see the archives for rec.aviation).

The first combat crews were trained in the US, then send to Thailand to fly
missions to North Vietnam.  They had a high loss rate for night TFR
missions.  Then they found out why:

The TFR was set to fly the plane at 200 feet.  The TFR couldn't see trees,
and some trees in SE Asia grow over 250 feet high...

                 Ron Wanttaja (ssc-vax!wanttaja)

Re: Results of a recent security review (RISKS 4.52)

"Arnold D. Robbins {EUCC}" <>
Wed, 11 Mar 87 12:59:30 EST
In article Risks 4.52 Andre Klossner writes on the licensing of OWNDIR.
>     [... and will someone sue AT&T if, after a license is duly obtained, a
>     devastating Trojan horse is perpetrated using this flaw/feature ?  PGN]

There has been a bunch of discussion about this in mod.os.minix; basically
within a year of the patent, it was released for Public Use, i.e. anyone
who wants to can use the setuid concept (which is why minix does). The
article there cited real U.S. Patent Office publications giving the details.
(I probably should have saved the article but didn't.) Anyway, I'm writing
to try and cut off the spread of misinformation as early as possible.

I find the moderator's point more interesting; the people to sue would be
the manufacturer who incorporated the feature, not AT&T who invented it...

Arnold Robbins
CSNET:  arnold@emory    BITNET: arnold@emoryu1
UUCP:   { akgua, decvax, gatech, sb1, sb6, sunatl }!emory!arnold

Risks of Maintaining RISKS

Peter G. Neumann <Neumann@CSL.SRI.COM>
Wed 11 Mar 87 11:56:14-PST
I received a complaint from Dave Parnas that he was suddenly receiving mail
intended for the automatic BITNET mail list maintainer.  It turns out that
two readers forgot, or did not read, the old instructions.  Sorry, there is
nothing I can do on BITNET to prevent it, although I make a big effort (but
still not foolproof) on CSL.SRI.COM.  (At least it hasn't happened here yet,
although I have received numerous retries to RISKS following rejected mail
inappropriately sent to the LIST.)  PLEASE READ THE MASTHEAD.  Reminder for
BITNETters, once again:

    For WISCVM, send mail to LISTSERV@CMUCCVMA, with a single line request:
  SUBSCRIBE MD4H your name         or        UNSUBSCRIBE MD4H your name
    For FINHUTC, send mail to LISTSERV@FINHUTC, with a single line request:
  SUBSCRIBE RISKS your name        or        UNSUBSCRIBE RISKS your name
    For UGA, send mail to LISTSERV@UGA, with a single line request:
  SUBSCRIBE RISKS your name        or        UNSUBSCRIBE RISKS your name

(All three may be work interchangeably — I'm not sure.)

Please report problems with the web pages to the maintainer