The Risks Digest

The RISKS Digest

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 1 Issue 33

Wednesday, 1 Jan 1986


oStar Wars and Bank of NY
Brint Cooper
Chris Hibbert
Jim Horning
o Lipton and SDI
Herb Lin
o The robot sentry
Martin Minow
o Murphy is watching YOU
Rob Austein
o Re: Failure probabilities in decision chains
Stephen Wolff

Re: Can Bank of New York Bank on Star Wars? [PGN's retitling]

Brint Cooper <abc@BRL.ARPA>
Mon, 23 Dec 85 17:38:18 EST
The idea of independent, non-communicating "battle groups" for an SDI
system sounds great.  But what about the "fratricide" problem?


Re: Can Bank of New York Bank on Star Wars? [PGN's retitling]

Mon, 30 Dec 85 12:00:25 PST
To: RISKS FORUM (Peter G. Neumann, Coordinator) <RISKS@SRI-CSL.ARPA>
cc: horning@decwrl.DEC.COM (Jim Horning)

  From: horning@decwrl.DEC.COM (Jim Horning)
  Date: 20 Dec 1985 1413-PST (Friday)
  Subject: Can Bank of New York Bank on Star Wars? [PGN's retitling]

  Last night in the debate at Stanford on the technical feasibility of the
  SDI, Richard Lipton chose the financial network as an example of the
  advantages of a distributed system (such as he is proposing for SDI)
  over a centralized one. "There have been no catastrophes."  [What about
  the ARPANET collapse?  PGN] 

The ARPANET collapse is a good contrasting case to show what Lipton was
talking about.  His point about the financial "network" is that it isn't
a monolithic system, but a set of many (dozens?, scores?, hundreds?)
independant systems.  Any one of the systems could fail (even
catastrophically) and it wouldn't be much of a problem for the whole
system.  ARPANet is a single monolithic system, centrally designed and
administered.  Most bugs manifest at exactly the same  provocations at
widely separated parts of the net.

There are at least a couple of separate national networks of automatic
teller machines, and if any one of them dies, it shouldn't have any
effect on the others, or on any of the banks with only local networks,
or no networks at all.  It would take a collapse of the phone system to
put them all out of commission. 

ARPANet on the other hand is a monolithic system.  There is one protocol
that all parts of the system must share, a common medium is used, and in
there are only a few implementations of the protocols.  It doesn't take
much to blow the whole system out of the water.  (For the most part it's
as reliable as it is is only because it gets constant use, and new
parts aren't put in until they are shown to work most of the time.)

What Lipton was proposing at the Stanford debate was that we make an
anti-missile shield from many separately designed and implemented parts
so that their failure modes are more independant.  This is a good idea,
and if it were done, I would have plenty of faith in the system.
However, that's not the way government gets things done.  Since the DOD
is running the program there's no way there would be more than three
"separate" designs, and they would all go through the same approval
process, removing many of the differences they started with.

Back to the ARPANet example, if you look at a larger system than just
ARPA, including UUCP, DECNet, IBM's internal network, as well as the
SOURCE, TYMNet, Compuserve, etc., you find the same robustness.  ARPANet
may die and be out of commission for a long time, and most people will
still be able to get work done through some other medium, since only a
fraction of the people using computer networks depend on any one of


Re: Can Bank of New York Bank on Star Wars? [PGN's retitling]

Jim Horning <horning@decwrl.DEC.COM >
30 Dec 1985 1419-PST (Monday)

I agree with many of your comments, and feel that the $38 billion
problem at Bank of New York is much more typical of how problems in
nominally "independent" systems can propagate because of the intrinsic
need to communicate. (As an example of a non-obvious interaction,
recall its effects on the platinum futures market.)

In addition to the problems you cite, Lipton's scheme suffers from a
few other flaws, including:

- The "simulation" that indicates that "only 5-10% extra bullets" would
be needed apparently makes two dubious assumptions:
    1) Independent "battle groups" (with sufficient "teraflops") can
    pinpoint targets as accurately as a cooperating distributed
    2) Each "battle group" is able to recognize all "kills" by
    any battle group. I.e., the "extra bullets" counted are only
    those that are fired simultaneously at a target. With many
    of the proposed weapons, targets would be disabled, rather
    than disintegrated; with kinetic weapons, a single target
    could disperse to form a threat crowd.
    (Note that observation of kills is a form of communication
    intrinsic to the problem.)

- There were good systems reasons (completely outside of the computing
requirements) that led the Fletcher commision to propose
cradle-to-grave tracking (especially for RV vs. decoy discrimination)
and a layered defense. Lipton gave no evidence of understanding
those reasons, let alone making credible alternate proposals.

- The systems that you cite, and that he cited, are all ones where each
component is in routine use under the exact circumstances that they
must be reliable for. No matter how many independent subsystems the
Lipton SDI is divided into, NONE of them will get this kind of routine
use under conditions of saturation attack where reliability will be
most critical. Thus there is a high probability that each of them would
fail (perhaps in independent ways!).

Jim H.

Lipton and SDI

Mon, 23 Dec 85 18:09:59 EST
To: horning@DECWRL.DEC.COM

    From: horning at decwrl.DEC.COM (Jim Horning)

    More generally, I am interested in reactions to Lipton's proposal that
    SDI reliability would be improved by having hundreds or thousands of
    "independent" orbiting "battle groups," with no communication between
    separate groups (to prevent failures from propagating), and separately
    designed and implemented hardware and software for each group (to
    prevent common design flaws from affecting multiple groups). 

That is absurd on the face of it.  To prevent propagation of failures,
systems must be truly independent.  To see the nonsense involved,
assume layer #1 can kill 90% of the incoming threat, and layer #2 is
sized to handle a maximum threat that is 10% of the originally
launched threat.  If layer 1 fails catastrophically, you're screwed in
layer #2.  Even if Layers 1 and 2 don't talk to each other, they're
not truly independent.

The robot sentry

<minow%rex.DEC@decwrl.DEC.COM >
Friday, 27 Dec 1985 13:11:28-PST
The following appeared on USENET net.general today (Dec 27).  Martin.

   "A much more sinister arrival on the robot scene is named Prowler.
Created by Robot Defense Systems in Colorado, Prowler has been designed
for use as a sentry to guard military installations, warehouses and
other sites where security is important.  When made available in the
near future, this squat, sturdy, mobile device will carry
microcomputers, software and sensors capable of locating intruders.
Chillingly, buyers will be able to arm Prowler with machine guns and
grenade launchers; they'll also be able to program the robot to fire at
will.  The manufacturer claims that interest in Prowler has been high,
both among domestic companies who see it as a comparatively low-cost
replacement for 24-hour human security, and certain foreign countries
where government officials might prefer guards that will never revolt."
                                        — US Air magazine
-- JP Massar, Thinking Machines Corporation, Cambridge, MA
-- ihnp4!godot!massar, 
-- 617-876-1111
Posted Fri 27-Dec-1985 16:08 Maynard Time. Martin Minow MLO3-3/U8, DTN 223-9922

Murphy is watching YOU

Rob Austein <SRA@XX.LCS.MIT.EDU>
Mon, 23 Dec 1985 16:45 EST
About six hours after sending that message about mailers [RISKS-1.32], I
found myself with the pleasant task of doing bit level reconstruction of
XX's MAILQ: directory with DDT, because the system had crashed while MMAILR
was in the middle of a disk transfer.  Talk about ironic postscripts....

Cheers, Rob

Re: Failure probabilities in decision chains

Stephen Wolff <steve@BRL.ARPA>
Mon, 23 Dec 85 17:33:15 EST
* IF the overall decision is correct if and only if all five
  sub-decisions are correct, and
* IF the sub-decisions are statistically independent, and
* IF the probability that each sub-decision is correct is 0.9,
* THEN the probability that the overall decission is correct is
  0.9^5 = .59049 (vide any textbook in probability)

which is *suspiciously* close to "59%".  But when Bill Walsh
mentioned this problem to me in LA he was adamant that this was
NOT the explanation he wanted.      -s

Please report problems with the web pages to the maintainer