The Risks Digest

The RISKS Digest

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 1 Issue 10

Thursday, 12 Sep 1985


o Weizenbaum, etc.; even if SDI worked....
John Shore
John McCarthy
o More on SDI reliability
Martin Moore

Weizenbaum, etc.; even if SDI worked....

12 Sep 85 11:11:13 EDT (Thu)
From: John Shore <epi-dc!>

It's tempting to respond to Weizenbaum by arguing against the general
proposition, "Don't do it if there's a way around it".  After all, should we
refuse to develop bullet proof vests and to equip police officers with them
just because a criminal might approach from behind and stab them in the ass?

Assuming that a proposed defensive system will work, the relevant question
is what is the cost of developing it compared to the cost of getting around

In the case of SDI, one should distinguish between defense against a few
missiles vs. defense against a massive attack.  Either defense would be
enormously expensive to develop.  If the goal of the attacker is to detonate
a few bombs (or threaten to do so), then it is obviously easier and cheaper
to get around SDI than through SDI.  Here, Weizenbaum is probably right.  If
the goal is massive or total destruction (including destruction of our
missile forces), then getting around SDI (assuming SDI works) does not
appear to be either easy or inexpensive.  Here, Weizenbaum is probably
wrong.  In this case, however, the premise is most likely also wrong.

Moreover, suppose that the premise is right — i.e. SDI works perfectly.  As
Parnas has pointed out, there's no way for anyone to establish this fact,
which shows the absurdity of arguments like "give us SDI and we will
dismantle our missiles".


John McCarthy <JMC@SU-AI.ARPA>
12 Sep 85 0057 PDT
To:   risks@SRI-CSL.ARPA    

    Some remarks of mine about SDI on Stanford BBOARD have been referred
to.  For the benefit of non-readers of that BBOARD, they mainly concerned
whether I, like Chris Stuart, should use the IJCAI platform to say something
about it.  I said nothing in my lecture, but in my press conference, added
to my remarks on AI, the remark that there was no principle of computer
science that says that programs of any particular task cannot be written and
debugged.  Not much interest was shown by the assembled press; there was
exactly one question on that point.

    At the suggestion of Robert Jastrow, who is one of the main
scientific defenders of SDI, I made the same point in letters to three
Congressmen, said to be influential in the matter of SDI appropriations.

    Now I shall say my opinion about SDI.

    1. If it can be done, it should.  If it affords complete protect,
that's great, and if it affords partial protection, that's good.  The
balance of terror is a bad thing.  Here are answers to some counter
arguments to its desirability.  (a) Joe Weizenbaum says that it attempts a
technological solution to a problem that should be solved morally.  Alas,
moral progress has been so slow that almost the only moral problems to be
even partially solved are those that can at least partially been turned into
technological problems.  For example, the technology of contraception has
greatly reduced human unhappiness.  (b) It is argued that the Soviets would
have to attack at the first sign of deployment.  Every past imminent advance
by either side has in principle given the other side some temptation to
strike before it can be deployed.  So far as we know, neither side has even
come close to giving in to such temptation.  One reason is that the effect
of any advance is always subject to a probabilistic estimate, so temporizing
has always looked better than attacking.  Even if SDI works very well, it
may be that no-one will be able to be sure that it is that good.

    However, most likely the main reason has been is that neither side
ascribes the very worst intentions to the other with certainty.  Each side
has always said, "Perhaps they don't actually mean to attack us.  Why have a
nuclear war for sure instead of only a certain probability?"  Anyway the
Soviets have experienced a period in which we had complete nuclear
superiority and didn't attack them.

2. My opinion is that if the physics of the problem permits a good
anti-missile defense the programs can be written and verified.  However, it
will be quite difficult and will require dedicated work.  It won't be done
by people who are against the whole project.  Computer checked proofs of
program correctness will probably play some role.  So will anticipating what
kind of bugs would be most serious and putting the biggest effort into
avoiding them.  Having many people go over and discuss all the critical
parts of the program will also be important.

More on SDI reliability

Tue, 10 Sep 85 13:56:45 CDT
Cc: soft-eng@mit-xx, lin@mit-mc, mooremj@eglin-vax

> From: Herb Lin <LIN@MIT-MC.ARPA>
> My primary complaint about your otherwise interesting table is that it
> assumes independent failure modes.  I think it is much more likely
> that the effects of coupled failures are larger.  In particular, given
> the failure of one platform, it is more likely that more than one will
> fail.

Good point.  My original post did concern only statistically independent
failures.  If I can be forgiven one more table, I'll address coupled failures.

Independent failures are caused by events isolated to a single platform,
e.g., electrical component failures.  The occurrence of such a failure in
platform J does not affect the probability of a similar failure in platform K,
i.e., P(K|J) = P(K|~J) = P(K).

Coupled failures are failures such that the probability of failure is low in
any platform, but is greatly increased in all platforms when it occurs in
any one of them.  For example, consider that a hostile power might develop a 
new method for its missiles to escape detection.  The probability that it will 
fool any one platform may be low; but if it fools one platform it is likely to
fool more than one, perhaps all.  For arbitrary platforms J and K, P(K|J) <>

The original false positive table is not affected by this, since it showed 
the probability that at least one platform would fail.  Coupled failures
do not change that probability, only the probability that if one fails, others 
will (although it is true that while this country might be able to explain
away a single false positive, explaining a whole bunch of them could be a lot 

The false negative case is where the kicker really comes in.  The original
false negative table applies to independent failures.  The following table
is structured similarly, but instead of using the probability of failure (Pn),
it uses the degree of coupling, Pn(K|J).  This table shows, for a 100-platform
system, the probability of various numbers of successful responses, given
that at least one system has experienced a coupled failure.

Pn(K|J):  .5      .6      .7      .8      .9      .95     .99
N:  0 | 1.0000  1.0000  1.0000  1.0000  1.0000  1.0000  1.0000
    5 | 1.0000  1.0000  1.0000  1.0000  0.9746  0.5550  0.0033
   10 | 1.0000  1.0000  1.0000  0.9973  0.5355  0.0265  0.0000
   15 | 1.0000  1.0000  0.9998  0.9123  0.0677  0.0001  0.0000
   20 | 1.0000  1.0000  0.9896  0.5200  0.0017  0.0000  0.0000
   25 | 1.0000  0.9993  0.8740  0.1204  0.0000  0.0000  0.0000
   30 | 1.0000  0.9822  0.5116  0.0097  0.0000  0.0000  0.0000
   35 | 0.9988  0.8525  0.1465  0.0003  0.0000  0.0000  0.0000
   40 | 0.9781  0.5054  0.0176  0.0000  0.0000  0.0000  0.0000
   45 | 0.8426  0.1574  0.0008  0.0000  0.0000  0.0000  0.0000
   50 | 0.5000  0.0219  0.0000  0.0000  0.0000  0.0000  0.0000
   55 | 0.1574  0.0013  0.0000  0.0000  0.0000  0.0000  0.0000
   60 | 0.0219  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000
   65 | 0.0012  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000
   70 | 0.0000  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000

For example, if the degree of coupling is 0.7 — that is, if something that
causes failure in one platform has a 70% chance of causing failure in any
other platform — then the probability is 51.16% that at least 30 of 100
platforms will respond correctly, 14.65% that at least 35 will, and so on,
that the probability that the first platform will fail is UNRELATED to the
probabilities in this table! 

As far as the relative probabilities of independent and coupled failures,
I haven't a clue.  The independent failures are the easiest to get a handle
on through reliability theory; the coupled failures may be the result of
unknown shortcomings in design, or due to unknown hostile actions.  (There
is an old saying that there are always more unknown errors than known errors,
because known errors are limited, but unknown errors are unbounded by 
                                            Martin Moore

Please report problems with the web pages to the maintainer