Please try the URL privacy information feature enabled by clicking the flashlight icon above. This will reveal two icons after each link the body of the digest. The shield takes you to a breakdown of Terms of Service for the site - however only a small number of sites are covered at the moment. The flashlight take you to an analysis of the various trackers etc. that the linked site delivers. Please let the website maintainer know if you find this useful or not. As a RISKS reader, you will probably not be surprised by what is revealed…
With RISKS-1.45, my MM saturated completely on its ability to cope with all
of the Volume 1 issues of the RISKS Forum (at 289 pages) in one .TXT file,
so it must be time to start Volume 2! Thus, let me take this opportunity to
review the six months since RISKS Volume 1 Issue 1, on 1 August 1985.
The SDI pallaver seems to have overwhelmed some of you, but it has opened up
some serious problems — as does the unreleased Eastport Report, which is
perhaps most interesting for what it does not cover! (You may have concluded
that SDI is a religious issue that cannot be decided rationally?) The Space
Shuttle tragedy also opens up all sorts of questions on how much trust we
can place in technology and what the risks are — although those questions
have been around all along. They are simply elevated to a higher level of
public awareness now (temporarily?). For example, the issue of the safety
of self-destruct mechanisms has been lurking, and is clearly a concern in
general — but is only one concern. There are lots of others. We must
remember that risks come many sources — not just from maliciousness and
accidents, but also from unforeseen combinations of problems.
So far we have addressed many important issues, although some of them quite
superficially. We have overlapped on occasions with ARMS-D, Soft-Eng, and
Security distributions. However, it seems that RISKS does provide a focus
that cuts across these other mailings, and that it serves a useful purpose
-- particularly when it indeed focuses on the risks to the public.
A few platitudes are in order:
No system is ever going to be 100% guaranteed all of the time,
especially if it runs standalone. Even with humans in the loop,
humans can make mistakes — especially in real-time.
Risks may come from unexpected directions. A system that has run
perfectly may still suddenly malfunction. Hardware may fail. Bugs may
remain undetected for years, and suddenly become activated. Besides,
software may age poorly, especially if changes somewhere else interfere
with old working software.
The operating environment may contain risks that undermine sound design,
impementation, and operations.
The notion that all critical concerns — security, reliability, etc. --
can be confined to a small portion of a computer system or distributed
system (e.g., a kernel) is a fantasy, particularly with conventionally
designed systems.
The notion that distributed control solves problems that cannot easily be
solved with central control is also often a myth — problems of updating,
synchronization, concurrency, backup, verifiability, etc., may be equally
severe in many operating environments.
Lowest-bidder efforts are intrinsically risky. Commercial software is
commonly far behind the state of the art. That may be an advantage in
some cases (!), but is often detrimental. But there are significant
benefits to be gained from using certain software engineering techniques.
There are inherent risks in using computer systems in critical
environments. These must be continually reexamined. Critical and open
discussion of critical systems and critical environments is essential.
We as technologists must better understand the risks and their implications.
And we must apply that increased understanding to new developments.
And now, a word from our sponsor: This forum was established at the request
of the Association for Computing Machinery, and chartered as an activity of
the ACM Committee on Computers and Public Policy (of which I am the
chairman). However, the opinions reflected herein do not constitute an
endorsement by the ACM. On the other hand, if I screw up and do something
the ACM does not like, I probably lose my (volunteer) job.
As a summary of some of the problems that have occurred, and as a possible
inspiration for undiscussed areas of concern or areas of hope, I include in
this issue an update of the disaster list that those of you who have been
with us since the beginning saw in Vol 1 No 1.
Peter
********************************************
SOME COMPUTER-RELATED DISASTERS AND OTHER EGREGIOUS HORRORS
Compiled by Peter G. Neumann (31 January 1986)
The following list is drawn largely from back issues of ACM SIGSOFT Software
Engineering Notes [SEN], references to which are cited as (SEN vol no), where
vol 11 = 1986. Some incidents are well documented, others need further study.
Please send corrections/additions+refs to PGNeumann, SRI International, BN168,
Menlo Park CA 94025, phone 415-859-2375, Neumann@SRI-CSL.ARPA.
Legend: ! = Loss of Life; * = Potentially Life-Critical;
$ = Loss of Money/Equipment; S = Security/Privacy/Integrity Flaw
3 mos unrepaired weather buoy; $1.25M award (SEN 10 5) [NY Times 13 Aug 85]
** SAC/NORAD: 50 false alerts in 1979 (SEN 5 3), incl. a simulated attack whose
outputs accidentally triggered a live scramble [9 Nov 1979] (SEN 5 3);
** BMEWS at Thule detected rising moon as incoming missiles [5 Oct 1960]
(SEN 8 3). See E.C. Berkeley, The Computer Revolution, pp. 175-177, 1962.
** Returning space junk detected as missiles. Daniel Ford, The Button, p. 85
** WWMCCS false alarms triggered scrams [3-6 Jun 1980] (SEN 5 3, Ford pp 78-84)
** DSP East satellite sensors overloaded by Siberian gas-field fire (Ford p 62)
** 747SP (China Air.) autopilot tried to hold at 41,000 ft after engine failed,
other engines died in stall, plane lost 32,000 feet [19 Feb 85] (SEN 10 2)
** 767 (UA 310 to Denver) four minutes without engines [August 1983] (SEN 8 5)
* F18 missile thrust while clamped, plane lost 20,000 feet (SEN 8 5)
* Mercury astronauts forced into manual reentry (SEN 8 3)
* Cosmic rays halve shuttle Challenger comm for 14 hours [8 Oct 84] (SEN 10 1)
* Frigate George Philip fired missile in opposite direction (SEN 8 5)
$ Hurricane Gloria in NY closes Midwest Stock Exchange (SEN 11 1)
$S Debit card copying easy despite encryption (DC Metro, SF BART, etc.)
$S Microwave phone calls easily interceptable; portable phones spoofable
$S Sputnik frequencies triggered garage-door openers
------------------------------ SOFTWARE -----------------------------------
!$ 1983 Colorado River flood, faulty data/model? Too much water held back
prior to spring thaws; 6 deaths, $ millions damage [NY Times 4 Jul 1983]
*$ Mariner 1: Atlas booster launch failure DO 100 I=1.10 (not 1,10) (SEN 8 5)
*$ Mariner 18: aborted due to missing NOT in program (SEN 5 2)
*$ F18: plane crashed due to missing exception condition, pilot OK (SEN 6 2)
*$ F14 off aircraft carrier into North Sea; due to software? (SEN 8 3)
*$ F14 lost to uncontrollable spin, traced to tactical software (SEN 9 5)
*$ El Dorado brake computer bug caused recall of all El Dorados (SEN 4 4)
$$ Viking had a misaligned antenna due to a faulty code patch (SEN 9 5)
$$ First Space Shuttle backup launch-computer synch problem (SEN 6 5 [Garman])
* Second Space Shuttle operational simulation: tight loop upon cancellation of
an attempted abort; required manual override (SEN 7 1)
* Second Shuttle simulation: bug found in jettisoning an SRB (SEN 8 3)
*$ Delays of two Discovery shuttle launches due to backup computer outage
[most recently 25 Aug 85] (SEN 10 5) [NY Times 26 August 1985]
* Shuttle STS-6 bugs in live Dual Mission software prevented aborts (SEN 11 1)
* Gemini V 100mi landing err, prog ignored orbital motion around sun (SEN 9 1)
* F16 simulation: plane flipped over whenever it crossed equator (SEN 5 2)
* F16 simulation: upside-down F16 deadlock over left vs. right roll (SEN 9 5)
* Nuclear reactor design: bug in Shock II model/program (SEN 4 2)
* Reactor overheating, low-oil indicator; two-fault coincidence (SEN 8 5)
* SF BART train doors sometimes open on long legs between stations (SEN 8 5)
$ IRS reprogramming delays; interest paid on over 1,150,000 refunds (SEN 10 3)
$ $32 BILLION overdraft at Bank of New York (prog counter overflow) (SEN 11 1)
*S Numerous system intrusions and penetrations; implanted Trojan horses; 414s;
intrusions to TRW Credit Information Service, British Telecom's Prestel,
Santa Clara prison data system (inmate altered release date) (SEN 10 1).
Computerized time-bomb inserted by programmer (for extortion?) (10 3)
PC Graphics program Trojan horse (ArfArf) wiped out users' files (SEN 10 5)
*$ Union Carbide leak (135 injuries) exacerbated by program not handling
aldicarb oxime plus operator error [NY Times 14 and 24 Aug 85] (SEN 10 5)
* Multipatient monitoring system recalled; mixed up patients (SEN 11 1)
* Pacemaker locked up when being adjusted by doctor (SEN 11 1)
* Diagnostic lab instrument misprogrammed (SEN 11 1)
S Chernenko at MOSKVAX: network mail hoax [1 April 1984] (SEN 9 4)
S VMS tape backup SW trashed disc directories dumped in image mode (SEN 8 5)
*$ C&P computer crashes 44,000 DC phones (SEN 1 1)
$ 1979 AT&T program bug downed phone service to Greece for months (SEN 10 3)
$ Demo NatComm thank-you mailing mistitled supporters [NY Times, 16 Dec 1984]
$ Slow responses in Bankwire interface SW resulted in double posting of tens
of $millions, with interest losses (SEN 10 5)
$ Program bug permitted auto-teller overdrafts in Washington State (SEN 10 3)
- Quebec election prediction gave loser big win [1981] (SEN 10 2, p. 25-26)
- Other election problems including mid-stream corrections (HW/SW) (SEN 10 3)
- SW vendor rigs elections? (David Burnham, NY Times front page, 29 July 1985)
- Alaskan DMV program bug jails driver [Computerworld 15 Apr 85] (SEN 10 3)
- Vancouver Stock Index lost 574 points over 22 months — roundoff (SEN 9 1)
- Gobbling of legitimate automatic teller cards (SEN 9 2, another SEN 10 5)
------------------------- HARDWARE/SOFTWARE --------------------------------
! Michigan man killed by robotic die-casting machinery (SEN 10 2, 11 1)
! Japanese mechanic killed by malfunctioning Kawasaki robot (SEN 10 1, 10 3)
[Electronic Engineering Times, 21 December 1981]
! Chinese computer builder electrocuted by his smart computer. (WWN headline:
"Jealous Computer Zaps its Creator" after he built newer one!!) (SEN 10 1)
* FAA Air Traffic Control: many computer system outages (e.g., SEN 5 3)
* ARPANET ground to a complete halt [27 Oct 1980] (SEN 6 1 [Rosen])
*$ Ford Mark VII wiring fires: flaw in computerized air suspension (SEN 10 3)
$S Harrah's $1.7 Million payoff scam — Trojan horse chip (SEN 8 5)
$ Great Northeast power blackout due to threshold set-too-low being exceeded
$ Power blackout of 10 Western states, propagated error [2 Oct 1984] (SEN 9 5)
$ NY Stock Exch. halted for 41 minutes; drum channel errors killed primary
and backup computer systems [24 Feb 72]
- SF Muni Metro: Ghost Train reappeared, forcing manual operation (SEN 8 3)
*$ Computer-controlled turntable for huge set ground "Grind" to halt (SEN 10 2)
*$ 8080 control system dropped bits and boulders from 80 ft conveyor (SEN 10 2)
S 1984 Rose Bowl hoax, scoreboard takeover ("Cal Tech vs. MIT") (SEN 9 2)
------- COMPUTER AS CATALYST, HUMAN FRAILTIES, OR UNKNOWN CAUSES ------------
!!$ Korean Airlines 007 shot down [1 Sept 1983], killing 269; autopilot left on
HDG 246 rather than INERTIAL NAV? (NYReview 25 Apr 85, SEN 9 1, SEN 10 3)
!!$ Air New Zealand crashed into mountain [28 Nov 1979]; computer course data
error had been detected and fixed, but pilots not informed (SEN 6 3 & 6 5)
! Woman killed daughter, tried to kill son and self; "computer error" blamed
for false report of their all having an incurable disease (SEN 10 3)
* Unarmed Soviet missile crashed in Finland. Wrong flight path? (SEN 10 2)
*$ South Pacific Airlines, 200 aboard, 500 mi off course near USSR [6 Oct 1984]
*S San Francisco Public Defender's database accessible to police (SEN 10 2)
* Various cases of false arrest due to computer database use (SEN 10 3, 11 1)
* Avionics failed, design used digitized copier-distorted curves (SEN 10 5)
$ .5M transaction became $500M, due to "000" convention; $200M lost (SEN 10 3)
$ Possible fraud on reinsurance — message time stamp faked??? (SEN 10 5)
$ N-step reinsurance cycle; SW checked only N=1 and 2 (SEN 10 5)
* FAA Air Traffic Control: many near-misses not reported (SEN 10 3)
!$$ Shuttle Challenger explosion, 7 killed. Cause not yet known. [29 Jan 86]
--------------- ILLUSTRATIVE OF POTENTIAL FUTURE PROBLEMS ------------------
*S Many known/past security flaws in computer operating systems and application
programs. Discovery of new flaws running way ahead of their elimination.
* Expert systems in critical environments: unpredictability if (unknowingly)
outside of range of competence, e.g., incompleteness of rule base. StarWars
$S Embezzlements, e.g., Muhammed Ali swindle [$23.2 Million], Security Pacific
[$10.2 Million], City National Beverly Hills CA [$1.1 Million, 23 Mar 1979]
[These were only marginally computer-related, but suggestive. Others
are known, but not publically acknowledged.]
--------------------- REFUTATION OF EARLIER REPORT -------------------------
* "Exocet missile not on expected-missile list, detected as friend" (SEN 8 3)
[see Sheffield sinking, reported in New Scientist 97, p. 353, 2/10/83];
Officially denied by British Minister of Defence Peter Blaker
[New Scientist, vol 97, page 502, 24 Feb 83]. Rather, sinking abetted by
defensive equipment being turned off to reduce communication interference?
[See also anecdotes from ACM Symposium on Operating Systems Principles,
SOSP 7 (SEN 5 1) and follow-on (SEN 7 1).]
Please report problems with the web pages to the maintainer