The Risks Digest

The RISKS Digest

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 6 Issue 40

Wednesday 9 March 1988


o First Boston faces substantial loss
Dave Curry
o Reliance on computers
o Number plates sans sense
Niels Kristian Jensen via Espen Andersen
o Re: New Macintosh virus...
David HM Spector
o [Psychological Aspects of] Safe Systems
Hugh Davies
o Re: Bank ATMs and checking your statements
Paul Fuqua
o Re: waning arithmetic skills; erroneous large phone bills
Toby Gottfried
o Trusting your calculator
Dan Franklin
o Calculator Self Test (was: Disappearing skills)
Mark W. Eichin
o Re: Disappearing skills
Bruce Hamilton
o Computer Ethics in the curriculum
Rodney Hoffman
o Database Correlation
Darin McGrew
o Info on RISKS (comp.risks)

First Boston faces substantial loss

Dave Curry <>
Wed, 09 Mar 88 14:56:33 EST
From Information Week, March 7, 1988, p. 8:

First Boston has confirmed that the company faces a substantial loss due to
inaccuracies in its system that inventories mortgage-backed securities.  The
company won't comment on the cause of the inventory error until an internal
audit is completed, but it's expected that losses will range between $10
million and $50 million.

Reliance on computers

Tue, 8 Mar 88 14:39 MST
Here is another entry on the danger of relying on the machine answer.

I have a friend who works at Inland Steel.  They control many of their furnaces
with special digital controllers made in the 60's.  They then have these
controllers hooked up to pdp-11 computers in a network to manage production and
quality control.  The 11's are responsible for lot tracking and special orders
while the controllers are charged with carrying out the production.  During a 2
week down spell one of the annealing furnaces was shut-down for repairs and was
allowed to cool.  (Usually furnaces are left idling even when not in
production) When it was ready to return to production a true cold start was
down to the furnace and it's controller.  At power up, a power supply burned
out in the digital thermometer so that it registered 0F.  The controller
knowing that it was in a cold start ordered 100% power to the gas tubes.  The
furnace should reach operating temperature within 1 hr, but with the dead
thermometer was registering no increase so the controller just waited for 4
hrs.  In the time period the furnace burned itself out doing several hundred
thousand dollars damage and will be out of service 6 - 12 months.  The amazing
thing was that 4 workers were there supervising the cold start and all of them
watched this knowing the computer was in control and must know what it was
doing even though their experience told them otherwise.

I see a few faults here in design.

     1) having a single fault failure path in the controller.

     2) the controller not having logic to recognize temp rise
        must occur rapidly in response to firing the burners.

     3) blind reliance on technology by skilled workers.

     4) no safety mechanisms external to the conttroller to recognize a
        runaway condition.

     5) No computer polling of sensors to test them response (self-test)

Number plates sans sense

Tue, 08 Mar 88 09:08:47 SET
Having read the discussion on number plate "mismatches", I would like to submit
the following.   Espen Andersen    Compuserve: 71361,3425    EDB85007 at NOBIVM

From NUTWORKS (The Electronic Humor Magazine) Issue No 13,
Submitted by Niels Kristian Jensen (C83821@NEUVM1.BITNET).

                              The plate.

In the U.S. it is possible to buy a 7-character "personalized" license
plate describing the car or the owner.  To apply, one must list 3
choices in order of preference.  If one's first choice has already
been assigned to another driver, one gets one's second choice, etc.
These facts and a computer system made the following possible:

Mr. X wanted a new plate.  He got the form and filled in his first two
preferences.  If these two possibilities had already been assigned to
someone else, Mr. X didn't want a personalized plate at all, so for
his third choice he wrote "NOPLATE".

           He got "NOPLATE" as his new personalized plate.

At first he got mad, but he mounted it on his car anyway.  Within a
week he had received offers for over 100 dollars for this ingenius
plate, and he began to grow partial to it.  In fact, he liked it.

His feelings changed by the end of the month, however, because by then
Mr. X had recieved nearly a hundred parking tickets.  Why?  Well, any
time a police officer spots a car missing a licence plate, he or she
will write "NOPLATE" on the ticket/citation.

The computer matched every single one of these with... Mr. X.

The computer program couldn't be changed, but the ways of the parking
guards could.  So don't *EVER* order the custom plate: "NONE".

(It would make you Mr. X the second)

    [This is an old warhorse.  I couldn't find it in my archives, 
    but I am sure it has been either in an early SEN or in RISKS.  
    On the other hand, recycling is big in California as well as
    vanity plates.  PGN]

Re: New Macintosh virus...

David HM Spector <spector@vx2.GBA.NYU.EDU>
Wed, 9 Mar 88 18:09:54 EST
(It may not have filtered all the way through the news system yet, but
here's what I wrote to comp.sys.mac and to Chris...)

...  There seem now to be (at least) 5 Macintosh viruses on the loose, two are
of the "Brandow/MacMag" variety and the rest are based on the sources I
described in Risks Digest some time ago.  Its gonna be a rough year..

  From: spector@vx2.GBA.NYU.EDU (David HM Spector)
  Date: 8-Mar-88 10:34 EST
  Subject: Re: I've got a virus and I don't like it
  Organization: New York University

It seems you have been bitten by a virus whose sources were uploaded to Compu-
serve sevral months ago...  The author, a fellow in West Germany, thought it
would be educational to distribute these example viruses in source form to
encourage people to write defenses against them.  His stated intent in writing
a virus in the first place was to keep people from running possibly virus
ridden program on their production Macintoshes which had been previously hit
by viruses....  its signature, in the orignal sources, was a resource type of
nVir... its a simple yet potent virus and very easily modified to do bad things.

    ... unfortunately the only way around most of these viruses is to
replace your system folder. (Make sure you do this from a WRITE-LOCKED 
copy of the Apple System installer... or else you'll end up back where you
started, with an infected system.... there is another problems, that being
that the virus  that was on CompuServe knows how to infect APPLICATIONS, as
well as the system itself.  Pretty depressing....

David HM Spector, NYU Graduate School of Business
AppleLink: D1161 CompuServe: 71260,1410 (212) 285-6080

Re: [Psychological Aspects of] Safe Systems

4 Mar 88 04:00:09 PST (Friday)
In RISKS 6.35, Nancy Leveson <> says;

  "It describes several serious incidents that occurred because pilots put too
  much confidence in automatic control systems, even to the extent of rejecting
  their own external evidence that the system was wrong"

I don't know about commercial pilot training, but I do know about PPL (Private
Pilots License) training, and I have read about military training, and when
learning to fly you are taught to ignore the 'evidence of your own eyes' and to
rely on instruments in all but the most ludicrous of circumstances.

Obviously this has the risk that the instruments may be wrong, but there are
many instances of pilots ignoring the instruments and relying on their senses
and subsequently doing stupid (or lethal) things. There is at least one
documented case of aircrew flying at night over water, seeing the lights of
fishing boats and believing them to be stars, inverting their aircraft, in
direct contradiction to what their instrumentation was telling them.  There is
also a fairly common phenomenon called 'the leans' when the pilot develops a
pronounced impression that the aircraft is not level, and rolls it in the
opposite direction to correct, even when the artificial horizon and the
subsequent handling of the aircraft tell him that the original roll was

In most cases, the (computer) user is not told to believe absolutely the
evidence of a machine over the evidence of his senses. But in the case of
aircraft he is explicitly trained to do so. This behooves us (as programmers,
etc.) to make sure that the machine is telling the truth!

Re: Bank ATMs and checking your statements [RISKS DIGEST 6.36]

Paul Fuqua <>
Mon, 7 Mar 88 18:56:20 CST
I had something similar happen about two years ago.  I made a $60 dollar
withdrawal, but the machine only gave me $40;  my receipt indicated $60.
When I went to the office next day to report it, the teller called over
a manager type.  Turns out they had been puzzling all morning over the
discrepancy in their records, which they reconcile daily, and were only
too happy to give me my $20.

It is important for *both* ends of a transaction to keep records.

Paul Fuqua, Texas Instruments Computer Science Center, Dallas, Texas
CSNet: or pf@ti-csl   UUCP: {smu, texsun, im4u, rice}!ti-csl!pf

Re: waning arithmetic skills; erroneous large phone bills (RISKS-6.35)

Toby Gottfried <decwrl!hplabs!felix!toby@ucbvax.Berkeley.EDU>
Tue, 8 Mar 88 13:08:30 pst
Regarding waning arithmetic skills:

Like pilots of high-tech planes who need to be able to tell when to observe
warning devices and when to override them, people using calculators need to
know when to trust the answers.  Even perfectly working calculators are
subject to input errors, and the consequences of a mistake can range from
inconsequential to disastrous.

To take a small example, once I was buying several different items at a drug
store.  The cashier keyed them in, and announced the total as $2.45.  This
was surprising to me, since the price of one of the items was $3.00
(apparently entered as $0.30).  Errors like this must happen frequently, but
this was so glaring it should have been noticed.

A case appropriate to this time of year:

It can be VERY important that adjusted gross income multiplied by 
tax rate (among other calculations) come out correctly.

Regarding the risks of erroneous large phone bills:

They occur infrequently, and they are normally fairly easy to correct, but
(with even lower probability), one COULD result in phone service being cut
off, just when there is a need to make an emergency call (fire, medical,
police).  Then the cost goes way up, as the reversibility goes way down.

Toby Gottfried    Costa Mesa, CA    FileNet Corp   {hplaps,trwrb}!felix!toby

Trusting your calculator

Dan Franklin <dan@WILMA.BBN.COM>
Wed, 9 Mar 88 13:21:35 EST
A cousin of mine is a police officer.  He is not at all computer-knowledgable,
but he understands very well that computers can make mistakes.  Why?  Because
of an exercise his class did when he was training.  Each member of the class
was given a calculator and told to do some specific arithmetic with it.  As it
turned out, half the calculators were (intentionally) defective, giving wrong
answers--but the class was not told this ahead of time; they had to find it out
the hard way.

I think this is a great idea.  It ought to be mandatory for people in every
profession in which non-computer-knowledgable people interact with computers
(especially government bureaucrats).  If my cousin is any indication, the
lesson will last a lifetime.
                                           Dan Franklin

Calculator Self Test (was: Disappearing skills)

Mark W. Eichin <eichin@ATHENA.MIT.EDU>
Tue, 8 Mar 88 23:29:09 EST
  >Maybe calculators should have some sort of "self-test" program built in 
  >which would be automatically invoked when the unit is powered up?
  >Al Stangenberger                    Dept. of Forestry & Resource Mgt.

HP-10 through 16 calculators have full self test of display (light up all
segments) and keyboard (you have to push all of the keys in an obvious order).
Interesting that I have never had either type of failure.
                                               Mark Eichin

    [I interpreted what was requested was a full self-test that includes
    computational checks of correctness!  PGN]

      True, I haven't seen that... but the original article mentioned an error
      due to a calculator with digits failing. I only wanted to point out
      that it there ARE groups (HP) that don't ignore the problem entirely.  

      The HP-41c automatically sanity checks the memory... I have not heard
      of any that actually check the numbers they generate, though
      documentation of what the self tests do is often poor.         _Mark_

Re: Disappearing skills

8 Mar 88 18:39:09 PST (Tuesday)
Can anyone point me to some good references on closed-cycle or
partly-open-cycle ecologies & economies?  I'm particularly interested 
in such questions as the minimal population and skill sets needed to 
construct something close to current US middle-class lifestyles.

Work on self-replicating factories might be of peripheral interest.  Earth
and space are both interesting, and probably have very different answers.

"Living off the land" as a 19th-century trapper or 20th-century bag lady is not
nearly as interesting as the more general problem of bootstrapping an economy.
Now perhaps I'm getting into third-world development issues.

Whoops, I'd better tie this into RISKS.  As the parts lose their grasp of the
whole (shared knowledge and values) and people become more specialized, I
suspect that more and more types of catastrophic breakdowns of civilization are
possible.  Mark Vonder Haar's comment about lost knowledge of farming is right
on.  Let us further consider that the agricultural 5% of the US don't "feed
themselves" -- they take out big loans, buy big machines, fuel, fertilizer,
COMPUTERS, and pesticides, and run specialized agri-factories.

Computer Ethics in the curriculum

Rodney Hoffman <>
9 Mar 88 10:51:59 PST (Wednesday)
'The Chronicle of Higher Education' for Feb. 24, 1988, page A15, carries a
lengthy article by Thomas J. DeLoughry with the headline


  Scanty attention to the topic seen as part of larger
  problem: Students know little about their profession

The article includes quotes from RISKS moderator Peter Neumann and from 
other RISKS contributors and names known to RISKS readers.  It includes 
discussions of programmer certification, computer science department 
accreditation, lack of research and administrative support for work in 
computer ethics, multi-disciplinary approaches, differences from other 
engineering disciplines, and details of the issue at MIT and Stanford.

Database Correlation (Re: RISKS-6.36)

Darin McGrew <ibmuupa!mcgrew@ucbvax.Berkeley.EDU>
Mon, 7 Mar 88 13:41:11 PST
In RISKS 6.36, Matt Fichtenbaum ( mentioned a cross correlation
between the databases of New York State's drivers' licenses and recipients of
aid to the blind.

The RISK is what is done with such a cross correlation, not that it was done
at all.  California allows drivers with clean driving records to renew their
licenses by mail.  I know one elderly individual (and have read of two others)
who continued to renew their licenses after attaining legal blindness, for
identification purposes.  The RISK is that people (eg, "The Bureaucracy")
might automatically prosecute such people without finding out what's really
going on.
                            Darin (I speak for myself, not for my employer.)

Please report problems with the web pages to the maintainer