The Risks Digest

The RISKS Digest

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 3 Issue 70

Monday, 29 September 1986


o Deliberate overrides?
Scott E. Preece
o Multiple causes and where to place the "blame"
o The Art of "Science" and its Computers
o No-lock Brakes
Peter Ladkin
o Sanity in Automating Keyword Abstracting
Brint Cooper
o The Network Is Getting Old?
o Info on RISKS (comp.risks)

Deliberate overrides?

"Scott E. Preece" <preece%ccvaxa@GSWD-VMS.ARPA>
Mon, 29 Sep 86 09:17:27 cdt
  /**** ccvaxa:fa.risks / LIN@XX.LCS.MI /  2:47 am  Sep 29, 1986 ****/
  > From: Charles R. Fry <Chucko at GODZILLA.SCH.Symbolics.COM>
  > > No matter how many automated controls we install on cars (and airplanes) 
  > > to prevent operators from exceeding their vehicles' limits, there will
  > > always be a need to allow the deliberate violation of these limits.  


  > This discussion about allowing overrides to programmed safety limits
  > worries me.

One of the nice things about computer-driven controls, as opposed to
mechanical controls, is that they allow you to be less draconian in
specifying limits.  You don't have to build a bomb release that can never,
ever allow the pilot to drop a bomb while inverted; you can instead say "You
know, if I do what you've asked, the bomb is going to fall on the wing and
probably strip off your starboard control surfaces." and the pilot can say
"Yes, I know, do it anyway."  And by providing a (safety-covered and
hard-to-reach) button that says "Override control limits" you can even make
it possible for the pilot to say in advance that at this point she feels the
danger in overriding the controls is smaller than the danger in not
overriding the controls.

The reason we think it's reasonable to require automated controls to allow
exceptions is that we know the automated controls have allowed and
encouraged us to incorporate limits and we recognize that (1) those limits
may have erred on the side of normal safety, (2) since the systems are new,
the necessary operational envelope may not be known, and (3) the interaction
of the limits may create unanticipated problems.  Yes, users should be
allowed to override automated controls in almost all cases AND designers
should make very, very sure that the effort to override is proportional to
the danger of the override.  In many cases there should also be logging of
overrides, so that operators, maintainers, and designers have an opportunity
to notice that actual use seems to be violating the design assumptions.

I wonder how many readers of this list [NO, this is NOT a survey, DON'T
write to tell me] drive cars with manual transmissions precisely because
they want to be in control, want to know that doing x and y will result in
the car doing z, without any control system in the way to place limits on
actions or responses...

scott preece  gould/csd - urbana  uucp: ihnp4!uiucdcs!ccvaxa!preece

Multiple causes and where to place the "blame"

Peter G. Neumann <Neumann@CSL.SRI.COM>
Mon 29 Sep 86 21:36:04-PDT
Today's AP noted that the FAA may cite the pilot of the Grumman Yankee for
being in restricted airspace, at precisely the moment of the crash between
the Aeromexico jet and the Piper Archer (which was also in restricted
airspace), which distracted the air traffic controller from attending to the
jet and the Piper (the absence of whose altitude information was also a
factor) -- at precisely the time the crash occurred.  The controller did
tell the Grumman pilot that he was in restricted air space, but then granted
him permission to continue (and that negotiation took two precious minutes
away from his attention to the jet).

The Art of "Science" and its Computers

Peter G. Neumann <Neumann@CSL.SRI.COM>
Mon 29 Sep 86 16:36:51-PDT
A computer of the AAAS sent out renewal bills for SCIENCE to some subscribers:
                 Subscription price $6647, Postage $732, 
                 Voluntary contribution $10, Total $5437.
The subscription price during 1986 was $60, and the accompanying letter from
the president of AAAS noted that inflation had required an increase.  A quite
amusing editorial by Daniel Koshland, Jr. in the 26 Sept 86 issue wondered 
whether any people would rush out to take advantage of the incorrect addition.

No-lock Brakes

Peter Ladkin <ladkin@kestrel.ARPA>
Mon, 29 Sep 86 14:47:16 pdt
A minor correction to Chuck Fry's comments - the first anti-skid system on a
production car was installed on a Jensen (pre-dating the Jensen-Healey) in
the 60s. It was made by Lockheed, and derived directly from aircraft systems.

Sanity in Automating Keyword Abstracting

Brint Cooper <abc@BRL.ARPA>
Mon, 29 Sep 86 15:09:02 EDT
Here is an example of a risk associated with the use of computers.  The risk
is to the accurate dissemination of information and is caused by faulty
programming (programmers?).

Today, the BRL Librarian informed us that the Defense Technical Information
Center (DTIC, formerly known as DDC) now requires that the titles of our
technical reports (the principal products of a research lab such as the BRL)
be written so that the "keywords" are found in the first five words of the

Thus, a report which formerly was titled "Communication Modeling in the
Artillery Control Experiment" with keywords "error control," "tactical
communications," "networks," and "modeling" would have to be titled
"Modeling, Tactical Communications, and Error Control Networks," thus
sounding like, as one chap here put it, "a four volume set by Harry van
Trees" instead of a 25 page report.

Exact text of our librarian's notice follows:

  > We have been advised by DTIC that the titles of technical reports should
  > be designed with the key words positioned in the first five words of the
  > title. This is because only the first five words are used in a title
  > search in the DTIC electronic data base DROLS.(DEFENSE RESEARCH ON LINE
  > SYSTEM).  Important to know (and remember) is that articles are counted
  > in those first five words.  Therefore a report entitled "A report of the
  > effect........." will not have any key words picked up in a title
  > search.  If you currently have a report in editing, we will review it
  > and if it it does not comply with the DTIC recommendation we will advise
  > you so that it can be reworked.  If you currently have a report under
  > review or in writing you might like to think about a title change.
  > Please give this widest possible dissemination.

ARPA:    UUCP:  ...{seismo,unc,,decvax,cbosgd}!brl-smoke!abc

Dr Brinton Cooper, U.S. Army Ballistic Research Laboratory
Attn: SLCBR-SE-C (Cooper),  Aberdeen Proving Ground, MD  21005-5066
Offc:    301-278-6883    AV:  298-6883     FTS: 939-6883   Home: 301-879-8927

                I started to add a diatribe, but gave up in annoyance.  PGN]

The Network Is Getting Old?

Peter G. Neumann <Neumann@CSL.SRI.COM>
Mon 29 Sep 86 09:50:22-PDT
There has been an enormous amount of difficulty in dealing with the ARPANET
since early September, perhaps related to the installation of new IMP
software and subsequent patches when the new release did not work properly.
I had devastating problems TELNETing from four different East-coast hosts to
three different SRI systems (irrespective of whether there was a gateway at
my end, and even with no loads on either system).  No answers have been
forthcoming from any of our gurus, so the problems remain pervasive and
painful.  I am also getting a rash of returned net-barfed RISKS mail (as
well as RISKS filling up peoples' directories when they go on vacation).
There are also local problems.  Last Friday a message from SRI-STRIPE to
SRI-CSL took 7 hours to be delivered, while TN and FTP between those two
machines worked fine.

  From: David L. Edwards <DLE@SRI-STRIPE.ARPA>
  It is becoming increasingly apparent that there are serious network
  problems.  There has been some discussion of this on the TCPIP forum.
  Network delays, failed connections, inability to make connections etc. are
  being reported by hosts all over the network.

  BillW has noticed that direct communications between SRI and SU are bad.  In
  your case there are one or two gateways involved in addition to the IMPs.
  The mailer recently reported 750+ attempted connections with 670+ failures.

Old age?  Software rot?  Incompatible changes to net software?  Saturation?
Who knows.  Stay tuned.

Please report problems with the web pages to the maintainer