The RISKS Digest
Volume 1 Issue 41

Sunday, 19th January 1986

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Please try the URL privacy information feature enabled by clicking the flashlight icon above. This will reveal two icons after each link the body of the digest. The shield takes you to a breakdown of Terms of Service for the site - however only a small number of sites are covered at the moment. The flashlight take you to an analysis of the various trackers etc. that the linked site delivers. Please let the website maintainer know if you find this useful or not. As a RISKS reader, you will probably not be surprised by what is revealed…

Contents

On a Clear Day You Can See Forever ... or Nothing At All
Peter G. Neumann
Unreleased SDIO Computing Panel Report: Specialists Fault `Star Wars' Work
Man in the loop and magnetic bottles
Jon Jacky

On a Clear Day You Can See Forever ... or Nothing At All

Peter G. Neumann <Neumann@SRI-CSL.ARPA>
Sun 19 Jan 86 16:29:27-PST
To: RISKS@SRI-CSL.ARPA

This issue of RISKS once again concerns the topic of SDI, both in its own
right and as being illustrative of a collection of risks from which we can
learn for other applications.  The problems that arise in considering SDI
are some of the most fundamental problems affecting our lives at the moment.
The news item that follows this message was probably seen by most of you who
read newspapers.  It is included here for the record, and for any of you who
missed it.

Occasionally I get a complaint that the RISKS Forum seems biased.  Some of
those comments have come from proponents of SDI.  I continually seek to
include both sides of any critical issue.  If only one side is willing to
speak out, RISKS may appear biased to those who do not want to get drawn
into the discussion.  That is unfortunate, particularly if one side can
eschew debate and then complain that the debate is rigged.  Oh, well, we are
damned if we do, and damned if we don't.  Our ACM sponsorship stated at the
outset of this Forum that we should make every attempt to be nonpartisan.  I
have on various occasions offered RISKS as an outlet to the SDI panel, but
have been rebuffed with "RISKS is biased".  Too bad.  It looks as if some
serious though went into the report that is described below, and it should
certainly have the benefit of exposure to the social process.  But if the
proponents of SDI are hiding under a protective shield other than the one
they are trying to create, then that is an anti-social process instead.

As I have noted earlier, it is certainly easier to criticize others than to
offer alternatives.  Generally, the proof of the pudding is in the eating.
Since the pudding has not jelled yet (indeed, it has not been cooked, which
is fortunate in that suitable recipes do not yet exist), discussion is based
on such things as dietary customs, a liking for untasted goodies, bad
experiences with the effects of overly rich food, or allergy to pudding.
De gustibus non disputandem est.

It is significant that software problems are explicitly identified in the
cited report as being critical to SDI.  (That conclusion was certainly
self-evident a priori, but its being emphasized will certainly point to many
problems that must be addressed — not just for SDI).  The risks therein are
enormous, and I hope that RISKS will objectively discuss them (along with
other risks) and what can be done to avoid them.  SOFT-ENG should also have
much valuable material on the techniques and role of software engineering.

However, as a meta-issue that is certainly relevant here, it is a tragedy of
our times that software engineering techniques are still not widely used by
the commercial developers of critical software.  Some of those developers
are literally many years behind the state of the art.  The fault is probably
distributed in many places, such as researchers who do not make their
research and development tools really amenable to realistic use; software
developers who take too many short-cuts and are too concerned with profits;
government administrators who do not insist on higher-quality software with
carefully preestablished requirements, and who do not follow through with
strict control measures and penalties; politicians who create and perpetuate
the bureaucracy; and the general public — which puts up with it all.  The
waste is enormous — not only in dollars, but in human resources and
misplaced efforts.  (I am referring here generically to a wide class of
developments, not to SDI alone.)

Peter


Unreleased SDIO Computing Panel Report

Walter Hamscher <hamscher@MIT-HTVAX.ARPA>
Sat, 18 Jan 86 12:07:27 est
From the Boston Globe, page 1, Saturday, 18 Jan 1986

         SPECIALISTS FAULT `STAR WARS' WORK

               By Fred Kaplan
                Globe Staff

WASHINGTON - In a report commissioned by the Pentagon's Strategic
Defense Initiative Office, a group of top computer software experts
concludes that the SDI office is going about the task of building a
`star wars' missile-defense system the wrong way.

The report does say developing a proper software program for SDI is
feasible.  However, it says the SDI office and its defense
contractors are assuming they can develop the `star wars' weapons
and sensors first and write its computer software afterward - when,
in fact, an effective defense will be impossible unless this order
is reversed.

The authors of the report, all avowed supporters of the SDI program,
met for 17 days last summer and held further discussions before
writing the report.  The report was submitted to the SDI office last
month, and has not been released publicly.

Software must be programmed to enable automatic communication
between the satellites that detect Soviet missiles and the SDI
weapons that will shoot the missiles down; between these weapons and
other sensors that can distinguish missiles from decoys and assess
whther the target was hit or missed; and between this entire network
and political authorities on the ground.  Hundreds of satellites,
battle stations, sensors, giant space mirrors and other devices
would be involved.  Computations must be made, and orders must be
given, in a matter of microseconds, with continuous updates and
revisions.

The report says all the various designs for strategic defense
systems proposed thus far demand "excessively sophisticated
software" that "cannot be adequately tested."  A design "that cannot
be tested ... is of no value," the report says.  And "excessively
complex software cannot be produced at any cost."

John Pike of the Federation of American Scientists, a critic of SDI
who did not serve on the panel that wrote the report, puts the
problem this way: "It's like buying a home computer first and then
discovering that the software you need won't run on it.  Or it's
like buying a Betamax and then discovering that your favorite movies
are only on VHS.

"This report," Pike continues, "says a lot of the money in the [SDI]
budget now is wasted because you'll end up buying the wrong
machines."

The report emphasizes that computer software programming is still a
young field with many unknown elements.  The report states, "The
panel expects no technological breakthrough that would make it
possible to write the celebrated `10 million lines of error-free
code,'" which SDI officials have acknowledged are necesary to make
the system, as currently envisioned, work.

Moreover, "there are no laws or formulae that can accurately predict
the successs or failure of a large software development."  Nor is it
possible today, the report says, to measure whether a software
program can be applied to an SDI battle-management system.

The report says these problems are not impossible to solve.
However, it says it will take at least two decades - and then only
if the organization of the program is radically changed.  Assuming
these fundamental uncertainties can be resolved, the report cites
other computer and software difficulties.  Among them:

* Flights of the space shuttle have frequently been delayed because
of computer problems found at the last moment.  Yet whereas the
shuttle's computers are designed to reain in operation for 1,000
hours without breaking down, the computers on board the satellites
used in an SDI system would have to be built to break down only once
every 100,000 hours.

David Parnas, a software specialist at the University of Victoria in
British Columbia, also says the experience of several shuttle
flights has allowed NASA to work out programming "bugs" over time.
"This kind of thing couldn't possibly work with SDI," he says.  "You
can't call the Russians in the middle of a war and say, `Wait a
minute, we have to recalculate some things.'"

Parnas was appointed a member of the panel that wrote the report.
However, he resigned a few weeks after its formation, saying its
work was pointless because SDI's software requirements were
impossible to fulfill.

Stephen Berlin of MIT's Laboratory for Computer Sciences [sic],
notes another difference between SDI and the shuttle: "The space
shuttle is not being shot at.  An SDI system almost certainly would
be."

* The system would be highly vulnerable not only to direct attack
but to nuclear weapons exploded in space as far as 1,000 kilometers
away.  "The high-energy neutron flux from a nuclear explosion is
expected to `erase' volatile semiconductor memory," the report says.
"Effective shielding is difficult."

The report recommends new ways of dealing with strategic defense
that organize the various components of an SDI system in a "loose
hierarchy," with tasks "delegated to and localized within parts of a
system."  Such a system would involve less complex and more testable
software, and could be adapted more easily to change.

The authors of the report - all software specialists at top
universities - acknowledge that it is not clear how to do all this,
and that the SDI office should "use independent contractors" who
could "tap the talent of leading researchers in the scientific
community," to study the problem further.


Man in the loop and magnetic bottles

Jon Jacky <jon@uw-june.arpa >
Sat, 18 Jan 86 16:37:24 PST
> (Dave Wade writes...)
> I ... maintained the control system for a series of Magnetic Fusion Energy
> experiments. The programs ran in "real time" and attempted to contain and
> shape a plasma. ... The point of this was that any failure of the control
> system was capable of killing personnel. ... The software was much smarter 
> than the operator and certainly swifter.   There was no way that I could
> hit the "kill" switch before the plasma got lose.

With all respect, I am having trouble accepting that there is not some 
misunderstanding or overdramatization in this account.  Are we to believe 
that people stood around where they could have gotten killed if the 
control system failed?  If this is true, the risks in question were certainly
unnecessary.  Why on earth were the operators not protected behind some kind 
of blast shield?  What assurances did they require to convince themselves that
the software was sufficiently bug-free that they could trust their lives with 
it?

Please report problems with the web pages to the maintainer

x
Top