The Risks Digest

The RISKS Digest

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 8 Issue 80

Friday 16 June 1989


o Disarmament by defect
Gerard Stafleu
o Even human-in-the-loop isn't foolproof. A test case.
Pete Holzmann
o Single point of failure? probably not.
Ephraim Vishniac
o Re: single point of failure -- Tokyo Stock Exchange
Patrick Wolfe
o Qantas Airliner Mishap
John Murray
o Theorem Proving by Computers
Tom Thomson
o Re: Computer electrocutes chess player ...
Dave Horsfall
Joel Kirsh
o Clerical error spares famed sex-fiend
Mike Albaugh
o Sabre computer problems revisited
Emily H. Lonsford
o Protection from Misdirected Radio Control Commands
Robert Horvitz
o Info on RISKS (comp.risks)

Disarmament by defect

Gerard Stafleu <>
Thu, 15 Jun 89 11:00:29 edt
We have seen quite a few articles on things going wrong with the
computerization of the military.  The latest example is the posting by Karl
Lehenbauer about "NORAD Computers: Years Late, Unusably Slow, $207 Million
Over Budget".

While most articles concerned the Western military, there is no doubt 
that our friends on the other side suffer from the same problems.  After 
all, they are doing their level best to get their hands on as much 
Western computer technology as possible.  (I have heard rumors that 
getting our technology to them is one of the most subtle and insidious 
plots developed by the CIA so far.)

As a result, it is reasonable to suspect that the advance of computer 
technology into the field of the military, has made it well neigh 
impossible to fight any war worth its SALT.  We find corroborating 
evidence for this position in the on-going disarmament proposals.  These 
have been started by Gorbachov, who knows an impossible situation when 
he sees one, and have now been taken over by Western leaders like Bush, 
who perhaps reads comp.risks.

So where the sheer incompetence of politicians and generals used to 
start wars, the sheer incompetence of us computer people has now put an 
end to it.  No mean feat.  For centuries humanity has been looking for 
the Weapon That Would End War Forever.  We have found it.  War has 
ended, not with the bang of a bomb, but with the gentle whisper of 
crashing software.

Gerard Stafleu, (519) 661-2151 Ext. 6043   BITNET address: gerard@uwovax

Even human-in-the-loop isn't foolproof. A test case.

Pete Holzmann <pete@slp.UUCP>
Wed, 14 Jun 89 14:10:29 PST
I was recently witness to an event that may be of interest to those
pondering safe user interfaces, man-in-the-loop questions, and the like. Not
being an expert in any of these areas myself, the only comment I'll make is
that it seems important to realize that there are cultural aspects of
human-technology interfaces. Never assume that a sane, well-trained person
will do 'the right thing'...

The following is a true story. No names are given, so as to protect the
participants from any further embarassment!

The scenario: experienced computer user/programmer needs to get some software
   mailed out during the weekend. He's relatively new to the office, so he has
   asked where the spare floppies are kept. He is told "there's a box with
   a bunch of floppies over on Joe's desk". There's a small error in these
   instructions: the correct box of spare floppies is on Jane's desk, not

What happens: He goes into the office alone on Saturday morning. Nobody
   is there to watch him (not that anybody normally would -- he's an
   expert, remember!) He finds no box of floppies on Joe's desk. But - aha! -
   there's a nice big box UNDER the desk. It is sealed. (It was delivered 
   the day before.) He opens the box, and finds a bunch of brand-new 
   commercial software packages. Shrink-wrapped, the whole bit.  Without 
   skipping a beat, he rips open a couple, and finds sealed white
   envelopes inside. The envelopes have your usual dire license agreement
   warning, beginning with a large STOP sign... ("STOP. Read before opening!
   etc...") Without skipping a beat, he rips open the envelopes, reformats
   the enclosed floppies, puts on new labels, and uses them to mail the
   software he needed. Thus ruining a few thousand dollars worth of new
   commercial software!

Now, before you read the answer, think about this puzzle: how could a sane
   person, an *expert* no less, completely ignore the warnings and do such
   a crazy thing? What are the RISKS implications of this?

Here's the answer:

He was able to do it, without even wondering whether it was the right thing
to do, because *in his experience*, what he saw and his resulting actions
were completely normal. In a previous job, his company received large
quantities of commercial software for evaluation and review. So much software
that they treated it like junk mail. The floppies were treated as reusable
media. With that in mind, his actions become completely reasonable! He was
trained to ignore dire warnings, expensive-looking software packaging, and
the like. The only thing of value in a box of commercial software, in his
experience, was the floppy disks themselves. And they were only useful once
reformatted and with fresh labels on them.


Pete Holzmann, Strategic Locations Planning     {hpda,pyramid}!octopus!slp!pete

single point of failure? probably not.

Thu, 15 Jun 89 09:47:53 EDT
In RISKS 8.79, Jerry Carlin (jmc@PacBell.COM) cites a story from 
the SF Chronicle (presumably San Francisco, and not some 'zine):

    The reporter quotes a story in "Manhattan, Inc" where it was
    disclosed that the main and backup computer for the Tokyo Stock
    Exchange sit right next to each other and in an area totally
    destroyed by the 1923 earthquake.

    This computer is the SOLE repository of Japan's offical records of
    stock ownership. Therefore if the computer is destroyed, all
    records of share ownership could disappear with obvious

It seems very unlikely that the computer is the SOLE repository.  More
likely, the two computers together with the on-site and off-site backups of
the data they contain are the widely distributed and highly redundant
repository of the stock ownership data.  That's not such an exciting story,
of course.

Supposing that Tokyo Exchange follows conventional backup procedures
(and they could easily do much better), destruction of both computers
would mean the loss of the current day's transactions; destruction of
the entire site might mean the loss of as much as one week's
transactions.  That's expensive, but it's not catastrophic.

Ephraim Vishniac, Thinking Machines Corporation, 245 First Street, 
Cambridge, MA 02142-1214

Re: single point of failure -- Tokyo Stock Exchange

Patrick Wolfe <>
Thu, 15 Jun 89 07:27:26 cdt
> This computer is the SOLE repository of Japan's offical records of stock
> ownership. Therefore if the computer is destroyed, all records of share
> ownership could disappear with obvious consequences.

This is why people in my position spend so much time with and are so concerned
about backups, so that the computer is not the "SOLE respository" of any
valuable information.  Well managed computer centers keep a set of complete
backups "offsite".  The ones with larger budgets use an storage location
complete with protection against fire and other environmental hazards.

The only story I have heard about a computer center that didn't keep any
backups is about US Cable in Lake County, IL.  Every six months or so, they
would unscramble all six pay channels for everyone for about a week, reportedly
because of a "computer problem" where they lost information about who was
paying for which channels.  If they had reliable backups, these records could
have been restored in a matter of hours, instead of a week.

        Patrick Wolfe   (, kailand!pat)
        System Manager, Kuck & Associates, Inc.

Qantas Airliner Mishap

John Murray <>
Thu, 15 Jun 89 17:30 PDT
I heard an NPR report recently about a Qantas plane going out of control
temporarily. It seems the autopilot suffered some sort of glitch. The (human)
pilot recovered from the dive, but several people bumped their heads, etc.
Since then, I've heard no follow-up, and seen nothing in comp.risks.

Was I hallucinating about the original report, or do I just have my head in a
bag this month??
                                        - John Murray, Amdahl Corp.

Theorem Proving by Computers

Tom Thomson <>
Thu, 8 Jun 89 09:43:04 bst
Henry Spencer comments on the acceptance by mathematicians of proof by
computer.  I think it's important to recognise that the computer introduces no
new risk here; we all believe group classification theorem, don't we, and
surely no-one has ever found time to check the proofs (or even understand the
underlying arguments) of all the lemmas and prior theorems involved therein.

Mathematics has a long history of "proofs" that aren't (eg the omission of
axioms about betweenness in geometry for a couple of thousand years); and quite
a few "theorems" have been disproved. Checking a proof is no easier than
checking a program. Checking that several proofs combine correctly to deliver a
new proof is no easier than checking that several programs combine correctly.

Do we have a new risk here - the risk that, because a computer is involved, we
will assume a new risk exists even when it doesn't (or is not new)?
                                                                    Tom Thomson

Re: Computer electrocutes chess player who beat it! (RISKS 8.75)

Dave Horsfall <munnari!!dave@uunet.UU.NET>
Thu, 8 Jun 89 11:21:29 est
[ Discusses receiving a strong shock from a 12-volt wiper ]

More likely he received an inductive shock from the electric motor.  There
is no way that a mere 12 volts will cause that sensation, but a kick of
a few hundred (thousand?) volts will do it, as the field collapses.

Computer electrocutes chess player

Joel Kirsh <KIRSH@UTORMED.bitnet>
Thu, 1 Jun 89 21:51:00 EDT
[Excerpted, from "Bioengineering: Biomedical, Medical and Clinical
Engineering", by A.T. Bahill (Prentice-Hall)]

     The impedance of the human body can be modeled as a core of low resistance
  (around 500 Ohms) ... and the skin with a higher resistance (1 to 100
  kiloOhms). ... the amount of electrical current necessary to induce
  venticular fibrillation [a "cardiac arrest"] in the human heart ... a minimum
  of 80 microAmps, 100 uA, and 180 uA [in three separate studies].

These values lead to estimates of the required voltage being anywhere from
240 mV (80uA times 3 kOhms) to 16 V (180 uA times 201 kOhms).  Of course,
this assumes that the current path crosses the chest.  Also, the heart is
especially susceptible to particular frequencies; good old 60 Hz is "the
optimum frequency for producing ventricular fibrillation." (Bahill)

Joel Kirsh, Faculty of Medicine, University of Toronto

Clerical error spares famed sex-fiend

Mike Albaugh <albaugh@dms.UUCP>
Wed May 31 10:54:17 1989
Quoting from Colin Wilson's "The Misfits":

    The revolutionary Marat decided that de Sade was a typical
    aristocratic libertine of the old regime and ought to die;
    by accident, however, he denounced the Marquis de la Salle,
    who was executed. Marat discovered his mistake and was about
    to rectify it when he was murdered in his bath by Charlotte
    Corday. Unaware of how close he had been to the guillotine,
    de Sade delivered an address describing Marat as a great man.

The parallels to modern wrongful arrest struck me, as well as the question
of how bad the reign of terror might have been with the "help" of modern
data processing. It appears the over-reliance on the accuracy of "official"
orders has been around for a while. Perhaps Madame DeFarge should have used
an error-correcting code in her knitting?

[My remembrance of early dp is that redundancy in the form of hash totals and
transaction serial numbers was used quite early, and seems to have been
forgotten, rather than enhanced, as we have "advanced"]

Sabre computer problems revisited

Emily H. Lonsford <>
Tuesday, 30 May 1989 10:01:46 EST
According to the May 22, 1989 issue of Computerworld, Sabre is run on 8
interconnected 3090-200E computers under a Sabre-modified version of ACP
(Airline Control Program OS by IBM).  A custom version of ACP has been used
there for about 20 years.  Neither ACP nor TPF 3.1 (due to be installed 3rd
qtr 89) provides the required protection, according to the article.  It seems
the errant 'core-walker' program modified another task that was formatting
disk drives - and the labels on 1080 disk drives were destroyed.

"The Sabre system is down an average of six minutes a week for maintenance,
Juracek noted, and is usually upgraded 'on the fly' so that service to other
parts of the world is not disrupted.  Because ACP cannot run without a disk
subsystem, Sabre software engineers took the unusual step of rebooting the
crashed system using IBM's VM operating system.  Then, they had to relabel each
disk drive and reset the pointers that indicate where passenger data is loca-
ted....While most Sabre data was not lost, the 'pointers' to all flight reser-
vation data were - and it took 100 programmers and systems engineers more than
10 hours to relabel each disk volume.  The system was restarted under ACP about
7 am CDT, and the reformatting was done by 11 am.  Then, due to pent-up network
demand, American's systems engineers had to gradually restart Sabre, slowly ad-
mitting more traffic from 27 front-end communications processors here."

ACP and TPF are IBM real-time operating systems that are designed to support
heavy transaction volumes.  The article goes on to state that virtual storage
will not be available under TPF until 1993.  Apparently other protection fea-
tures are not there either, such as private address spaces and multiple
storage protection keys, which are implemented under MVS.

Emily H. Lonsford, MITRE - Houston W123  (713) 333-0922

Protection from Misdirected Radio Control Commands

Robert Horvitz <rh%well%apple@sun.UUCP>
Fri, 2 Jun 89 00:45:43 pdt
In RISKS 8.75, MIchael Berkley quoted a newspaper article about an accident in
northern Ontario in which a radio-control signal intended for one mining
machine triggered an unintended response in a second machine, which pushed a
miner to his death.  Berkley asked:  "What kind of safeguards are possible in
this situation and are the safeguards reliable?"

I am not familiar with Canadian regulations for radio control, but they are
probably similar to US regulations.  As it happens, the FCC has just adopted
new rules governing radio signals from unlicensed devices, including radio
control systems (Gen. Docket 87-389:  First Report & Order adopted 30 March
1989).  The Commission is explicitly trying to encourage the proliferation of
low-power unlicensed radio devices of all types, in the spirit of
"deregulation" promoted by outgoing FCC Chairman Dennis Patrick.  The primary
feature of the new "Part 15" rules is to loosen restrictions on the use of
radio links in appliances and systems sold publicly.  The new rules begin to
take effect on June 23rd.  They are sure to lead quickly to a rash of new
products such as wireless modems, wireless VCR/camera units, new remote
monitoring and control systems for the home, etc.  One aspect of the new rules
relevant to the mining story is that the FCC set no maximum power limit for
radio emissions in mines, caves or tunnels.

A traditional feature of all "Part 15" devices is that they enjoy no right of
protection from interference - either from similar devices or from licensed
transmitters.  Licensing confers the right of non-interference.  Radio control
systems are generally unlicensed.

Since most of the services that the Commission regulates are for communication,
they are used to thinking of interference in terms of, e.g., degradation of TV
picture quality.  They are not used to thinking of it in terms of misdirected
control.  In fact, because Part 15 devices have no recognized right of
non-interference, the Commission's attitude is - and has always been - "buyer
beware/you're on your own."

Thus, the only safeguards we can expect in the US, to avoid accidents like the
one that killed the Canadian miner, are those voluntarily adopted by
manufacturers.  Fortunately, there is a relatively simple fix to the problem:
have each radio command begin with an identifier specifying which device is
being addressed, and have the identifier be unique enough that there is little
chance of two devices with the same identifier being co-located.  Better, have
the owner or operator be able to set the identifiers in the field, to ensure
each is unique within the transmitted signal's radius.

Over a dozen petitions have already been filed objecting to the FCC's new
rules.  I will probably be filing comments soon on behalf of the Association of
North American Radio Clubs.  I may raise this issue of radio control safety in
my filing.  But I'm sure the Commission will say that this is a matter for the
marketplace can decide, and no "interference" from them is needed.

Please report problems with the web pages to the maintainer