Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 1: Issue 17

Friday, 27 Sep 1985

Contents

SDI debate announcement

Minor risk to the pocket book

Eugene Miya

Social Impacts of Computing: Graduate Study at UC-Irvine

Rob Kling

Friendly enemy test teams

John Mashey

More protocol goofs

Dave Curry

SDI debate announcement

<genrad!teddy!lkk@mit-eddie.MIT.EDU>

26 Sep 1985 17:48-EST

To: risks@sri-csl.arpa

Dear Colleague,

Computer technology plays an ever-increasing role in the arms race.
The Strategic Defense Initiative (`Star Wars') is a leading example
of a military system in which almost complete reliance will be placed
on computerized decision making.  The feasibility and desirability of
this system are currently undergoing serious public debate.

On Monday, the 21st of October, at 8:00 pm in M.I.T.'s Kresge Auditorium, the
M.I.T. Laboratory for Computer Science and Computer Professionals for Social
Responsibility are co-sponsoring a public forum designed to raise many of the
technical issues surrounding the Strategic Defense Initiative (SDI).

Professor Michael Dertouzos, Director of the M.I.T. Laboratory for Computer
Science, will moderate a debate on the feasibility of software development
for the SDI project.  Dr. Danny Cohen (Director of the Systems Division at
the University of Southern California's Information Sciences Institute and
Chairman of the Strategic Defense Initiative Organization panel on Computing
in Support of Battle Management (SDIO/CSBM)) and Professor Charles Seitz
(Professor of Computer Science, California Institute of Technology and
member, SDIO/CSBM)) will speak in favor of the SDI proposal.  Professor
David Parnas (Lansdowne Professor of Computer Science, University of
Victoria, Canada) and Professor Joseph Weizenbaum (Professor of Computer
Science, Massachussetts Institute of Technology) will take the position that
such a software development project is infeasible.

Professor Parnas' resignation from the SDIO/CSBM panel in June of
this year, and the set of memos he wrote discussing the infeasibility
of the SDI project attracted extensive press coverage in June of this
year.

CPSR will be holding a reception for the speakers at La Groceria (853
Main Street in Cambridge) between 6:30 and 7:30.  Please join us for
dinner and an opportunity to meet some of the panelists.  The $25 per
plate donation will help us cover expenses for this forum and related
projects.  Please RSVP to Mark Vilain at (617) 648-4325.

Earlier that afternoon, the M.I.T. Technology and Culture Seminar
will sponsor a talk by Dr. James Ionson, Director of the Office of
Innovative Science and Technology for the SDIO.  After Dr. Ionson
describes the general research goals of SDI, two M.I.T. professors
will respond with varying views on why they have chosen to accept or
refuse funding for research from the SDIO.  A student representative
will report on student reaction to Star Wars projects on campus.
This talk will be held at MIT in building 9, room 150 at 4:00 p.m.

Minor risk to the pocket book

Eugene Miya <eugene@AMES-NAS.ARPA >

26 Sep 1985 1652-PDT (Thursday)

Here is a minor man-machine risk which occurred today (9/26) in the Silicon
Valley at El Torito [popular Mexician food chain] at lunch.  We arrived for
a late lunch and found that our bill was 50% over what appeared in the menu.
The cash register [one of those computer systems where they press buttons
rather than prices] was running on the dinner menu prices rather than the
lunch menu prices.  Since we arrived late, everybody else [e.g., hundreds of
people] were over-charged for lunch that day and perhaps earlier.  This
implies several things about the way customers in Si Valley treat their
bills [rich?  no verification? ...]  What about the restaurant?

From the Rock of Age Home for Retired Hackers:

--eugene miya,   NASA Ames Research Center,   eugene@ames-nas.ARPA

Re: Technology and Morality

Benjamin Thompson <munnari!mulga.oz!bjpt@seismo.CSS.GOV >

Fri, 27 Sep 85 12:01:18 EST

Organization: Computer Science, University of Melbourne

Nicholas Spies writes [in RISKS-1.13]:

  >             ...                 Hitler opened the Pandora's Box of applying
  >high-tech to warfare and it worked (at least until a higher-tech response
  >prevailed).

Technology has been successfully applied to warfare for millenia.  Alexander
the Great didn't win through having a bigger army (he didn't); he had better
weapons (e.g. he found out that ballistae propel spears faster than people do).
(and he had better trained soldiers etc. - education and technology basically).

  >       After WWII a new era was born in which global political power no
  >longer rested on moral authority but on a command of the new applied
  >sciences and scientists.

Nothing could be further removed from morality than the way in which global
political power is grabbed and maintained.  It has always been based upon
physical strength (which usually, but not always, corresponds to technological
strength).  History is a never-ending sequence of examples.

Social Impacts of Computing: Graduate Study at UC-Irvine

Rob-Kling <Kling%UCI-20B@UCI-ICSA>

26 Sep 1985 0920-PDT

To: risks@SRI-CSL
                                CORPS:
                        Graduate Education in
            Computing, Organizations, Policy, and Society
               at the University of California, Irvine

     This graduate concentration at the University of California,
Irvine provides an opportunity for scholars and students to
investigate the social dimensions of computerization in a setting
which supports reflective and sustained inquiry.

     The primary educational opportunities are PhD concentrations in
the Department of Information and Computer Science (ICS) and MS and
PhD concentrations in the Graduate School of Management (GSM).
Students in each concentration can specialize in studying the social
dimensions of computing.

     The faculty at Irvine have been active in this area, with many
interdisciplinary projects, since the early 1970's.  The faculty and
students in the CORPS have approached them with methods drawn from the
social sciences.

     The CORPS concentration focuses upon four related areas of inquiry: 

 1.  Examining the social consequences of different kinds of computerization
     on social life in organizations and in the larger society.

 2.  Examining the social dimensions of the work and organizational
     worlds in which computer technologies are developed, marketed,
     disseminated, deployed, and sustained.

 3.  Evaluating the effectiveness of strategies for managing the
     deployment and use of computer-based technologies.

 4.  Evaluating and proposing public policies which facilitate the
     development and use of computing in pro-social ways.


     Studies of these questions have focussed on complex information
systems, computer-based modelling, decision-support systems, the
myriad forms of office automation, electronic funds transfer systems,
expert systems, instructional computing, personal computers, automated
command and control systems, and computing at home.  The questions
vary from study to study.  They have included questions about the
effectiveness of these technologies, effective ways to manage them,
the social choices that they open or close off, the kind of social and
cultural life that develops around them, their political consequences,
and their social carrying costs.

     CORPS studies at Irvine have a distinctive orientation -

(i) in focussing on both public and private sectors,

(ii) in examining computerization in public life as well as within
      organizations,

(iii) by examining advanced and common computer-based technologies "in
      vivo" in ordinary settings, and

(iv) by employing analytical methods drawn from the social sciences.


         Organizational Arrangements and Admissions for CORPS

     The CORPS concentration is a special track within the normal graduate
degree programs of ICS and GSM.  Admission requirements for this
concentration are the same as for students who apply for a PhD in ICS or an
MS or PhD in GSM.  Students with varying backgrounds are encouraged to apply
for the PhD programs if they show strong research promise.

     The seven primary faculty in the CORPS concentration hold appointments
in the Department of Information and Computer Science and the Graduate
School of Management.  Additional faculty in the School of Social Sciences,
and the program on Social Ecology, have collaborated in research or have
taught key courses for CORPS students.  Our research is administered through
an interdisciplinary research institute at UCI which is part of the Graduate
Division, the Public Policy Research Organization.

Students who wish additional information about the CORPS concentration
should write to Professor Rob Kling (Kling@uci-icsa), Department of
Information and Computer Science, University of California, Irvine, Irvine,
Ca. 92717, 714-856-5955 or 856-7548, or Professor Kenneth Kraemer
(Kraemer@uci-icsa), Graduate School of Management, University of California,
Irvine, Irvine, Ca. 92717, 714-856-5246.

Friendly enemy test teams

John Mashey <mips!mash@glacier >

Wed, 25 Sep 85 01:17:53 pdt

John McCarthy <JMC@SU-AI.ARPA> writes in RISKS-1.10:

  > 2. My opinion is that if the physics of the problem permits a good
  > anti-missile defense the programs can be written and verified.  However, it
  > will be quite difficult and will require dedicated work.  It won't be done
  > by people who are against the whole project.  Computer checked proofs of
  > program correctness will probably play some role. So will anticipating what
  > kind of bugs would be most serious and putting the biggest effort into
  > avoiding them.  Having many people go over and discuss all the critical
  > parts of the program will also be important.
  
Perhaps the best way to make it work WOULD be to have a test team of
people (who might be skeptics, at least) trying to break it.  Most large
complex projects that actually worked, at least that I've seen, have
succeeded at least partly because they had a large test team who didn't
believe anything worked until it could get past the worst of their tests.
I don't know what the ratio is elsewhere, but many complex ATT/BTL projects
allocated 30-50% of the staff to building test frameworks designed to stress
the system under test.  Consider the recent history of evaluation of
new military systems (like the Sergeant York).  It's very hard for the
builders of something to evaluate it well; you need a good enemy for that.

     [Tiger teams have indeed had some success in finding the more obvious
      program bugs, but in general many flaws may remain.  This topic has been 
      raised superficially in past issues.  Perhaps we are ready for some
      detailed discussions on the strengths and limitations of testing.  PGN]

More protocol goofs

Dave Curry <davy@purdue-ecn.ARPA >

Thu, 26 Sep 85 22:29:41 EST

    [The original message contained 8576 characters, almost exclusively
     headers.  I have pruned it to give just the flavor.  PGN]

I'm forwarding this as a wonderful example of protocols getting
completely hosed... this mail of mine bounced for some unexplained
reason.  I resent the message the same day this came back, and it
went through just fine.

Looking at the headers should make the problem more than obvious...

--Dave Curry

Return-Path: <decvax!mcnc!unc!unc!unc!unc!unc!unc!unc!unc!mailer-daemon>
Received: from ee.ECN by ec.ECN; Thu, 6 Sep 84 19:48:01 est (4.12/5.20)
From: decvax!mcnc!unc!unc!unc!unc!unc!unc!unc!unc!mailer-daemon
Received: by ee.ECN; Thu, 6 Sep 84 19:47:35 est (4.12/5.20)
Return-Path: <decvax!mcnc!unc!unc!unc!unc!unc!unc!unc!unc!mailer-daemon>
Received: by decvax.UUCP (4.12/1.0)
    id AA24764; Thu, 6 Sep 84 02:25:55 edt
Received: by mcnc (4.12/4.7) id AA00613; Wed, 5 Sep 84 18:51:54 edt
Original-From: <unc!unc!unc!unc!unc!unc!unc!unc!mailer-daemon@mcnc>
     [... and so on iteratively, ad nauseum, down to... ]
Received: by unc (4.12/4.7) id AA06045; Wed, 5 Sep 84 09:07:16 edt
Original-From: <mailer-daemon@mcnc>
Received: by mcnc (4.12/4.7) id AB15479; Wed, 5 Sep 84 08:01:18 edt
Date: Sat, 1 Sep 84 10:03:51 est
Original-From: Mail Delivery Subsystem <MAILER-DAEMON@mcnc>
Subject: Returned mail: Unable to deliver mail
To: mcnc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!pur-ee!davy@unc

   ----- Transcript of session follows -----
554 sendall: too many hops (30 max)

   ----- Unsent message follows -----
Received: by mcnc (4.12/4.7) id AA15479; Wed, 5 Sep 84 08:01:18 edt
From: <mcnc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!pur-ee!davy@unc>
Received: by unc (4.12/4.7) id AA03055; Wed, 5 Sep 84 07:04:54 edt
Original-From: <mcnc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!unc!mcnc!pur-ee!davy@mcnc>
   [...]

   [I am reminded of the tale of the unattended British power station
    equipped with an automatic calling unit to report troubles.  When it
    finally had to dial the emergency reporting number, it received a
    recorded message that the number it had dialed was not valid.  PGN]

Report problems with the web pages to the maintainer