The Risks Digest

The RISKS Digest

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 3 Issue 13

Thursday, 26 June 1986


o The Risky Gap Between Two Design Cultures
Jack Goldberg
o Risks of nuclear power
Dan Franklin
o Research programs that pay for themselves
Rich Cowan
o Having an influence from "within the system"
Rich Cowan
o RISKS in running RISKS — continued
PGN and an unhappy Mailer
o Info on RISKS (comp.risks)

The Risky Gap Between Two Design Cultures

Wed 25 Jun 86 12:01:12-PDT
Over the centuries of experience in dealing with hazards, mechanical and
civil engineers developed a culture of safe design, with principles and
practices appropriate to the various kinds of products.  This culture was
expressed in the design of mechanisms that implemented various safety
functions, such as barriers to undesired motion, redundancy in the event of
local failures, self-adjustment to losses of tolerance, and so on.  For each
kind of product, particular mechanisms were developed to accomplish these
functions, e.g., pawls, detents, rails, ratchets, fuses.

The advent of computers and inexpensive sensors and motors made possible
tremendous economies in manufacture by eliminating all those particular
mechanisms and their often costly assembly (consider the dramatic comparison
in complexity of mechanism between the original teletype machine and a
modern typewriter/printer).  Mechanical design of the new systems has been
dramatically simplified, and the complex functions, including safety
functions, have been relegated to a control program.  In a sense, the design
is created on a blank slate.

Who creates that design?  Generally someone who is a professional
programmer, often a novice, who has inherited the culture of that
profession.  There are many aspects to that culture, but it rarely includes
the lore and practices of safe design (and the exposure to the machinery of
legal liability) that is the inheritance of mechanical and civil engineers.
It is often based on a partitioning of responsibility between the
hypothetical (and often anonymous) "customer" and the programmer-supplier, a
partitioning that hides the ultimate users from the designer.  Also, too
often, the programmer's education in matters of the physical world has been
compromised by the demands of training for his profession.

Often, the practitioners in the new culture see themselves as generalists,
able to solve any new problem, and they move frequently from one application
area to another.  Consequently, they seldom have the time to study and
understand the things that users or designers in a particular field know or
assume to be obvious, and so they must imagine and re-invent them.
Tragically, those imperfectly mastered things sometime seriously affect safety.

In short, the culture of safety that traditional engineers have expressed in
particular mechanisms has been tossed out along with those mechanisms, and
is being re-discovered, painfully, by a new generation of designers that has
no connection with the traditional culture.  In this light, risks arising in
contemporary computer-based system design may be seen as a consequence of a
gap between two design cultures.  The gap is both generational and
professional; there are many safety engineers in industry, but they and
programmers speak different languages.

In a different context, awareness of the loss of knowledge by experts in
various practices, due to their lack of replacement in the work force, has
stimulated some computerists to try to capture that knowledge.  How well
they are doing that is another matter, but it may be that some conscious
gap-bridging between the cultures would save the world some amount of
misfortune and misery.

                                    Jack Goldberg, SRI International

Risks of nuclear power

Dan Franklin <>
Tue, 24 Jun 86 14:03:42 EDT
TMPLee@DOCKMASTER.ARPA discusses nuclear energy vs. solar energy and "taking
all the risks into account".  The risk he is primarily concerned with is the
risk of falling off a solar energy device while cleaning off the snow.

If we are going to take all the risks into account, let's face it: the risks
to those involved in the actual energy production are simply insignificant
in the debate on nuclear vs. other forms of energy.  That debate focuses
almost entirely on the risks to innocent bystanders.  These are the risks
that always matter most, precisely because people do not willfully undertake
them, but rather end up subjected to them, and people are not willing to be
*subjected* to nearly as much risk as they are willing to *decide* to take,
or let others decide to take.

The fundamental political problem of nuclear power is that it has a small
probability of being disastrously more injurious to bystanders than any
other form of power generation except dams.  (Solar power satellites which
deliver their power by microwave are a future contender.)

TMPLee's mention of low-level radiation emitted by coal-fired plants is, of
course, directly relevant to this issue.  But in the wake of Chernobyl, as
in the wake of TMI (and in the wake of Pilgrim's safety problems...), the
small probability of disaster clearly needs to be discussed.

    Dan Franklin

Research programs that pay for themselves

Richard A. Cowan <COWAN@XX.LCS.MIT.EDU>
Thu 26 Jun 86 00:08:21-EDT
Let me add a few comments to Bob Estell's point #6:
 >   "Going to the moon in the '60's cost the USA nothing!...
 >   The DIFFERENCE between tax dollars paid by those wearing pacemakers, and 
 >   the "aid to their families" that would have been paid had those heart 
 >   patients died or been disabled, is more than $25 billion." 

  There are two problems with these types of conclusions.  First of all,
there are plenty of big non-space or non-military government programs that
we could spend our money on that are equally likely to have spinoffs; there
must be a reason why SDI should be built rather than these projects.  But
the classification barriers of SDI will inevitably reduce spinoffs.  Not
only that, but some things in SDI will certainly be useless commercially.
Pacemakers don't need to survive nuclear explosions.

  Secondly, any government program has an opportunity cost which is not
factored into your calculation: when we devote scientific resources to the
private sector, we lose out on the benefits we would have gained if those
resources weren't used up by the government.  An example is mentioned in a
May issue of the weekly trade paper "Electronics News": a Japanese witness
at some hearings on US competitiveness points out that the United States
spends hundreds of millions on high-strength, lightweight carbon materials
for aircraft wings, while the Japanese developed the same materials very
cheaply for golf clubs and tennis racquets.

   Are there things which we could use more than we could use SDI?  Are
there other government expenditures (perhaps national health insurance)
that would REALLY cost nothing?  Well, I recently heard that aside from
public police forces and the military, about $300 billion per year is spent
on security (including locks, alarms, etc.)  To get an idea where this
comes from, consider that MIT's police force costs a couple million, and
all universities put together must spend about $1 billion.

Now if businesses instead spent $100 billion of this money on raising the
minimum wage $2, spent $50 billion on reducing unemployment by reducing the
work week, and $20 billion went to the government to improve housing
programs and public facilities to keep young people occupied, then perhaps
the need for so much security would be reduced, because the root causes of
crime would be diminished.  It would therefore "cost nothing" for the
private sector to divert $170 billion of its security bill and improve the
social stability and welfare of the country.  The problem with such a plan
is that the benefits come only in the long term; only the greater short
term costs are seen on corporate balance sheets.


Having an influence from "within the system"

Richard A. Cowan <COWAN@XX.LCS.MIT.EDU>
Thu 26 Jun 86 00:11:07-EDT
And now a few comments on Bob Estell's point 10 on working for SDI:
> "But if we do take the opportunity, then we can use the
> managers' short term interests to an advantage; i.e., we can honestly say
> that "Star Wars" [R2D2 et al] is not possible today; and then diligently
> work to produce what is reasonable."

You have here touched upon what I believe is — more often than not — a
delusion:  that it is more effective to work within the system to change
it than to protest it from without.  In this case, working within the
system means working on Star Wars to demonstrate part of it to be feasible
or infeasible.

There are several problems with this.  First, within a large institution
you may be isolated from resources, or a diversity of viewpoints needed to
make an impartial decision.  This is less true with Star Wars than with
other programs because there's lots of mainstream publicity.  It is also
less true in a university than in a defense contractor.

Second, and more importantly, what an engineer says is likely to get
manipulated for political reasons — like the ignored warnings before the
space shuttle disaster.  If of 10,000 engineers working on SDI, 5000
include negative critical material in their research reports, and the other
5000 are completely uncritical of SDI, what do you think Congress will
hear?  Well, I can guarantee that they will hear mostly glowing reports
about research progress from upper-level managers and lobbying
organizations of the companies doing SDI research.  If your strategy
to change things is to become one of those upper-level managers, you may
have to compromise your values to achieve promotion, and temper your
criticisms to avoid losing "credibility" once you get there.

Yet Congress is hearing the other side on SDI.  How?  Because engineers are
not relying on the companies they work for to communicate their insight.
They are going outside the normal channels of communication — like the
1600 scientists working at government labs who recently petitioned Congress
to curtail SDI spending.  And ultimately, communicating one's concerns
directly to people in the community is necessary.

What is unfortunate, and I believe dangerous in a democracy, is that
people working for the government are afraid of speaking out on public
policy issues for fear of reprisal.  The recent statements by
Undersecretary of Defense for Research and Engineering Donald Hicks may
have heightened this fear.  Fortunately, the Pentagon has recently
dissociated itself from Hicks' statements.  (Science, May 23)


Returned mail: Service unavailable

Mail Delivery Subsystem <>
   [One of the greatest annoyances in running a large mailing enterprise such
    as RISKS is fielding the incessant net-barfs, including having my mailbox 
    cluttered with multiple copies of the Forum on net addresses that don't 
    work now and then.  (Some mailers that keep retrying periodically, and 
    send back advisories each time.)  Here is a fine example — which of 
    course more generally represents another type of risk in distributed 
    systems.  PGN]

    [FOOTNOTE:  The more general problem of copious rejected mail would be
    an order of magnitude worse if we went to individual messages rather
    than the current digest format.  (I now have requests from BITNET and 
    USENET to do send out undigestified messages, and would love to let them
    do the undigestifying.  Perhaps more regional reforwarding centers would 
    minimize rejects and reduce mailing list maintenance substantially.  But,
    I nevertheless get mystery rejection notices for people not even on my
    list, because of redistribution problems elsewhere.)]

   ----- Transcript of session follows -----
<>> DATA
<<< 554 <malloy>... Mail loop detected
<>> QUIT
<<< 554 sendall: too many hops (30 max)           [... but quite a brew-haha!]
554 <malloy@hull>... Service unavailable: Bad file number

   ----- Unsent message follows -----
Received: from pacific.ARPA by (4.12/ 1.1)
    id AA00971; Tue, 24 Jun 86 03:04:28 pdt
Received: from (hull.ARPA) by pacific.ARPA (4.12/4.7)
    id AA11313; Tue, 24 Jun 86 03:04:01 pdt
Received: from (aegean) by (2.2/SMI-2.0)
    id AA14974; Tue, 24 Jun 86 02:50:15 pdt
Received: from pacific.ARPA by (4.12/ 1.1)
    id AA00967; Tue, 24 Jun 86 03:04:07 pdt
Received: from (hull.ARPA) by pacific.ARPA (4.12/4.7)
    id AA11309; Tue, 24 Jun 86 03:03:40 pdt
                        [... many hops omitted ...  I think you get the idea.  
                             Notice the clock drift while you're at it.]
Received: from (hull.ARPA) by pacific.ARPA (4.12/4.7)
    id AA11281; Tue, 24 Jun 86 03:01:05 pdt
Received: from (aegean) by (2.2/SMI-2.0)
    id AA14942; Tue, 24 Jun 86 02:47:19 pdt
Received: from pacific.ARPA by (4.12/ 1.1)
    id AA00935; Tue, 24 Jun 86 03:01:10 pdt
Received: from (aegean.ARPA) by pacific.ARPA (4.12/4.7)
    id AA11271; Tue, 24 Jun 86 03:00:39 pdt
Received: from SRI-CSL.ARPA ( by (4.12/ 1.1)
    id AA00926; Tue, 24 Jun 86 03:00:30 pdt
Date: Tue 24 Jun 86 01:41:53-PDT
From: RISKS FORUM    (Peter G. Neumann, Coordinator) <RISKS@SRI-CSL.ARPA>
Subject: RISKS-3.12 [...]

     [But, gee Mr. Wizard, it worked just fine on the previous issues!]

Please report problems with the web pages to the maintainer