The Risks Digest

The RISKS Digest

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 8 Issue 14

Tuesday 24 January 1989


o Re: Medical Software -- testing and verification
Dave Parnas
o NSA and the Internet
Vint Cerf
o Re: Losing systems
Geoff Lane
o Computing Projects that Failed
Dave Platt
o Re: Object Oriented Programming
Benjamin Ellsworth
o Losing Systems
Henry Spencer
o Computer Emergency Response Team (CERT)
Brian M. Clapper
o Probability and Product Failure
Geoff Lane
o Probabilities and airplanes
Robert Colwell
Mike Olson
Dale Worley
o Info on RISKS (comp.risks)

Re: Medical Software (Are computer risks different?) (RISKS-8.7)

Dave Parnas <>
Mon, 23 Jan 89 07:48:13 EST
In his contribution to Risks 8.7 Jon Jacky makes the statement that the
problems of testing and verification are broadly similar whether the machine
includes a computer or not.  I have heard this argument from many "old-time"
engineers and consider it quite false.  In the testing of conventional (analog)
devices we make use of the fact that the functions are continuous and that one
can put an upper bound on the derivatives or the frequency spectrum.  We use
those mathematical properties in deciding how many tests are required to
validate a device.  When digital technology is involved there are no limits on
the rate of change.  Further, with digital technology, the number of tests
required for black-box testing increases sharply with the lowest known
upper-bound on the number of states in the device.  If we do "white-box"
testing, we can reduce the number of tests required by exploiting the
regularity of the state space.  In practice, the regularity is present and
helpful for the testing of hardware but not terribly useful for software
testing.  In short, the technology being used does make a big difference in
testing and validation.

While I agree with Jon's statement that industry practices in software
development are often much worse than for other kinds of technology, that is
not the only explanation of our "special problem".  The technology itself is a
great contributor and always will be.

David Parnas, Queen's University, Kingston Ontario

NSA and the Internet

23 Jan 1989 01:11-EST
John Gilmore asks why NSA has 5 IMPs if they are NOT monitoring the
Internet. So far as I know, NSA does not have 5 IMPs on the Internet. It has
one to support Dockmaster. The agency has a variety of internal networks, of
course, but none are likely to be linked to the Internet since they are used
for classified applications for which the Internet is not approved.

Does Mr. Gilmore have some evidence he wishes to present that suggests the
NSA is engaging in an unacceptable activity on the Internet?
                                                                  Vint Cerf

Re: Losing systems

"Geoff. Lane. Tel UK-061 275 6051" <ZZASSGL@CMS.UMRCC.AC.UK>
Mon, 23 Jan 89 09:34:58 GMT
In my experence the single most probable cause of a software project failing is
that the people who started the project have no real idea what they want in the
end. Almost everything else can be coped with but when you have to deal with a
constant stream of "design changes" not even the best people with the best
equipment can succeed.
                                        Geoff. Lane, UMRCC

Computing Projects that Failed

Dave Platt <>
Mon, 23 Jan 89 10:28:18 PST
On the subject of computing projects that failed for one reason or
another: I recommend that interested Risks readers look up some of Bob
Glass's books on this subject.  Glass has collected quite a number of
case-studies, changed the names to protect the innocent [and the
guilty, too], and organized them into categories according to the
primary reason for the failure (immature technology, wrong technology,
mismanagement, misimplementation, politics, etc.).  Some of the stories
are roaringly funny... f'rinstance, the mainframe at "Cornbelt U." that
survived a series of mishaps during installation (including being watered
by the University's lawn sprinklers), only to end up destroying itself
(and most of the building) during an earthquake.

Glass has written half-a-dozen books on the computing industry (most of
them date back to the '70s and early '80s).  The three most applicable
to Risks issues are: "Computing Projects that Failed", "Computer Messiahs:
More Computing Projects that Failed", and "Computing Catastrophies".
[I may be off a bit in the exact wording of the titles;  my copies are
at home.]

Based on the recent contributions to Risks concerning recent software-
project failures, it sounds to me as if most of the pitfalls that Glass
wrote about back in the '70s are alive and well in the late '80s!

Dave Platt    FIDONET:  Dave Platt on 1:204/444        VOICE: (415) 493-8805
  UUCP: ...!{ames,sun,uunet}!coherent!dplatt     DOMAIN:
  USNAIL: Coherent Thought Inc.  3350 West Bayshore #205  Palo Alto CA 94303

Re: Object Oriented Programming

Benjamin Ellsworth <>
Mon, 23 Jan 89 13:54:59 pst
Recently a professor from the local university taught a class on OOP at our
site.  During the first lecture, he said that via OOP one can add
functionality to the module without changing the code.  I asked
incredulously, "Without changing *any* code?"  He said, "Yes."  A manager at
the class sagely nodded his head.

I should hope the risks are obvious.

re: Losing Systems

Tue, 24 Jan 89 02:20:32 -0500
>The losing systems almost always contain some elements of newness; in fact on
>close inspection they usually contain several such elements...

To quote from John Gall's SYSTEMANTICS:  "A complex system that works is
invariably found to have evolved from a simple system that worked."  So
perhaps it's not so surprising that a lot of these done-yet-again-from-
scratch systems (how many different county records systems does the
world NEED?!?) fail.

                                     Henry Spencer at U of Toronto Zoology

Computer Emergency Response Team (CERT)

Brian M. Clapper <clapper@NADC.ARPA>
Tue, 24 Jan 89 10:18:01 EST
Excerpted from UNIX Today!, January 23, 1989 (reprinted without permission)

WASHINGTON -- The federal government's newly formed Computer Emergency
Response Team (CERT) is hoping to sign up 100 technical experts to aid in
its battle against computer viruses.
     CERT, formed last month by the Department of Defense's Advanced
Research Project Agency (DARPA) ... expects to sign volunteers from
federal, military and civilian agencies to act as advisors to users facing
possible network invasion.
     DARPA hopes to sign people from the National Institute of Science and
Technology, the National Security Agency, the Software Engineering
Institute and other government-funded university laboratories, and even the
     The standing team of UNIX security experts will replace an ad hoc
group pulled together by the Pentagon last November to deal with the
infection of UNIX systems allegedly brought on by Robert Morris Jr., a
government spokesman said.
     CERT's charter will also include an outreach program to help educate
users about what they can do the prevent security lapses, according to Susan
Duncal, a spokeswoman for CERT.  The group is expected to produce a
"security audit" checklist to which users can refer when assessing their
network vulnerability.  The group is also expected to focus on repairing
security lapses that exist in current UNIX software.  To contact CERT, call
the Software Engineering Institute at Carnegie-Mellon University in
Pittsburgh at (412) 268-7090; or use the Arpanet mailbox address

Probability and Product Failure

"Geoff. Lane. Tel UK-061 275 6051" <ZZASSGL@CMS.UMRCC.AC.UK>
Mon, 23 Jan 89 09:17:33 GMT
Unfortunately, from reports here in Britain after the M1 plane crash, it
appears that there is a real problem with "Common Mode" failures in aircraft
engines. So if one fails then the probability of a second failing during the
same flight is much higher than would be expected.  The probabilities of
failure are not independent.

(BTW - in "fly by wire" systems they attempt to avoid common mode errors in the
software by having three independent groups implementing the system on three
different types of processor. Firstly this does NOT eliminate the problems of
errors in the system specification from which all three designs are derived.
Secondly what happens 10 years later when the software is updated to
incorporate new developments - are three more independent software houses
commissioned to produce the new software - or would this be done in-house by
some part-time students?)
                                           Geoff Lane UMRCC.

Probabilities (Re: RISKS-8.12)

Robert Colwell <mfci!colwell@uunet.UU.NET>
Sun, 22 Jan 89 14:20:00 EST
There is a definite danger to this analysis, stemming mostly from its essential
correctness.  There was a plane within the last two years (if memory serves)
that lost all three of its engines on a flight precisely because such events
are not necessarily independent.  Turned out that the same mechanic had worked
on all three and made the same mistake on all three (left off an oil seal, I
think).  Another example is the nuclear reactor fire of a couple of years ago,
where all the redundant control wiring was for nought because somebody routed
them all through the same conduit, so they were all destroyed at the same time.

One must be extremely careful with abstract analyses like these -- they can
be seductive, and they can lead to unjustified conclusions.

real discrete probability and airplanes

Mike Olson <mao@blia.UUCP>
23 Jan 89 10:05:05 PST (Mon)
as at least two people have pointed out, my analysis of the likelihood of
failure was wrong.  i claimed that the probability of two engines failing
out of three was 6(p**2); the correct answer, of course, is 3(p**2).

thanks to A. Lester Buck (siswat!buck) and for
pointing out my error in a way befitting the kinder, gentler nation we
now live in.  it's not quite clear what i was computing, but it certainly
wasn't probability.  it wasn't even conditional probability, since i got
the independence argument wrong.

it's important to remember one of the real risks of the network -- the
potential for embarassing yourself in front of hundreds (thousands?) of
intelligent people.  next time, i check my work.
                    mike olson, britton lee, inc.

   [Also noted by Mike Wescott (m.wescott@ncrcae.Columbia.NCR.COM) and
   Dale Worley.  PGN]


Dale Worley <worley@compass.UUCP>
Mon, 23 Jan 89 10:11:29 EST
Actually, given that the probability of an engine failing during the trip,
year, etc. is p, and the probability of it not failing is q = 1 - p, then:

the probability of 0 engines failing is q**3
the probability of exactly 1 engine failing is 3 p q**2
the probability of exactly 2 engines failing is 3 p**2 q
the probability of all 3 engines failing is p**3

Given (we hope!) that p is very small, q is essentially 1, then p <>
p**2 <> p**3, so we can approximate:

  The probability of (at least) 1 engine failing is 3 p .
  The probability of (at least) 2 engines failing is 3 p**2 .

The trouble with "the probability of one engine failing is ... and the
probability of one of the remaining two failing is ..." is that is
double-counts the failures, for instance the probability of engine A failing,
*then* engine B is approximately 1/2 p**2, not p**2 as assumed by the previous
poster -- the other 1/2 p**2 times, engine B fails before engine A.

Dale Worley, Compass, Inc.                      compass!

Please report problems with the web pages to the maintainer