The Risks Digest

The RISKS Digest

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 5 Issue 69

Friday, 4 December 1987

Contents

o Can you sue an expert system?
Barry A. Stevens
o Risks of Portable Computers
PGN
o Beware the Temporary Employee
Howard Israel
o Truncated anything
Doug Mosher
o An ancient computer virus
Joe Dellinger
o Cable violations of privacy
Bob Rogers
o Re: Computer-controlled train runs red light
Steve Nuchia
o VM systems vulnerability
Doug Mosher
o Baby monitors end up 'bugging' the whole house
Shane Looker
o F4 in 'Nam (Re: Reversed signal polarity...)
Brent Chapman
o IRS computers (yet again!)
Joe Morris
o Journal of Computing and Society
Gary Chapman
o Info on RISKS (comp.risks)

Can you sue an expert system?

<portal!cup.portal!Barry_A_Stevens@Sun.COM>
Thu Dec 3 21:34:16 1987
I am interested in the legal aspects of using expert systems.

Consider, and please comment on, this scenario.

                     * * * * * * * * * * *

A well-respected, well-established expert systems(ES) company constructs
an expert financial advisory system. The firm employs the top ES
applications specialists in the country. The system is constructed with
help from the top domain experts in the financial services industry. It
is exhaustively tested, including verification of rules, verification of
reasoning, and further analyses to establish the system's overall value.
All results are excellent, and the system is offered for sale.

Joe Smith is looking for a financial advisory system. He reads the sales
literature, which lists names of experts whose advice was used when
building the system. It lists the credentials of the people in the
company who were the implementors. It lists names of satisfied users,
and quotes comments that praise the product. Joe wavers, weakens, and
buys the product.

"The product IS good,", Joe explains. "I got it up and running in less
than an hour!" Joe spends the remainder of that evening entering his own
personal financial data, answering questions asked by the ES, and
anticipating the results.

By now, you know the outcome. On the Friday morning before Black Monday,
the expert system tells Joe to "sell everything he has and go into the
stock market." ESs can usually explain their actions, and Joe asks for
an explanation. The ES replies "because ... it's only been going UP for
the past five years and there are NO PROBLEMS IN SIGHT."

Joe loses big on Monday. Since he lives in California, (where there is
one lawyer for every four households, or so it seems, and a motion
asking that a lawsuit be declared frivolous is itself declared
frivolous) he is going to sue someone. But who?

     The company that implemented the system?

     The domain experts that built their advice into the system?

     The knowledge engineers who turned expertise into a system?

     The distributor who sold an obviously defective product?

Will a warranty protect the parties involved? Probably not. If real
damages are involved, people will file lawsuits anyway.

Can the domain experts hide behind the company? Probably not. The
company will specifically want to use their names and reputations as the
source of credibility for the product. The user's reaction could be,
"There's the so-and-so who told me to go into the stock market."

Can the knowledge engineers be sued for faulty construction of a system?
Why not, when people who build anything else badly can be sued?

How about the distributor -- after all, he ultimately took money from
the customer and gave him the product.

                     * * * * * * * * * * *

I would be very interested in any of your thoughts on this subject. I'd
be happy to summarize the responses to the net.

Barry A. Stevens, Applied AI Systems, Inc., PO Box 2747, Del Mar, CA 92014
619-755-7231


Risks of Portable Computers

Peter G. Neumann <NEUMANN@csl.sri.com>
Thu 3 Dec 87 09:13:14-PST
Northwest Airlink had a real computer crash.  A 50-pound personal computer
belonging to a passenger, Ron Olstad, fell from a cargo pod on a Jet Stream
31 commuter plane landing in Oshkosh, Wisconsin.  The computer crash landed
into Ronald Miller's backyard.  Northwest Airlink replaced the computer.
[but not the divot!]

From the International Herald Tribune, 18 November 1987, p.3, courtesy of
Vicki Almstrum of Philips in Jarfalla (with `"' over the first two `a's).


Beware the Temporary Employee

Howard Israel <HIsrael@DOCKMASTER.ARPA>
Thu, 3 Dec 87 11:09 EST
BLUE CROSS ISSUES RED-FACED APOLOGY                  (Bergen Record, 12/2/87)

UPI, Providence, R.I.- A mysterious prankster has struck Blue Cross and Blue
Shield of Rhode Island and about 100 of its subscribers.  The subscribers
recently received letters from the health insurer concerning doctor bills, with
a postscript that would confirm the worst fears of many subscribers.  It read,
"Note: Your eligible charges will remain in file until hell freezes over."

Red-faced Blue Cross officials apologized to subscribers for the blooper but
have not been able to find the culprit.  A temporary employee who had access
to the computer that generated the letter left the day the sentence was added,
a spokesman said Monday.


truncated anything

Doug Mosher <SPGDCM%cmsa.Berkeley.EDU@ucbvax.Berkeley.EDU>
Thu, 03 Dec 87 18:02:20 PST
Following the discussions of truncating UNIX keywords to 8 characters without
notice, I would like to make the following statement/opinion/recommendation:

  It is ALWAYS BAD PRACTICE to delete anything without notice.

One special case of this is:

  It is ALWAYS BAD PRACTICE to truncate anything without notice.

Many examples over the years occur to me; here's a small partial list.

 --------

 Several products provide a really gimpy form of "Artificial Intelligence"
    by allowing the user to insert meaningless words, and ignoring
    them without notice.

    Example: syntax would be: "print amount by state by city".
    Syntax also allowed: "please print me a report showing the
     amount sorted by state and then by city".

 This starts off looking good, but ends up deleting all sorts of necessary
 keywords if you ever slip and misspell them, with very hazardous outcomes. You
 get a report, all right; no messages are issued; you go ahead and buy
 things/fire people/explode bombs, based on the WRONG REPORT RESULTS.

 Products which do this include FOCUS and TELL-A-GRAF.

 --------

 Some Microsoft BASIC interpreters give the impression of allowing variable
 names longer than 2 characters, but actually they just truncate to two
 characters. This results in VERY PAINFUL and hard to find bugs, when one
 finally calls two things PRINT and PRACTICE or whatever.

 --------

 The IBM Pl/I compiler truncates variable names to 31 characters. This is
 longer than most people are motivated to use so it has rarely caused anyone
 pain. External names are truncated to 8 characters, but at least notice is
 given.

 --------

 IBM MVS dataset names are truncated at 44 characters. This is not a frequent
 source of problems, but many folks have tripped over this at one time or
 another. To compound things, under various circumstances in the MVS operating
 system, dataset names are further shrunk to lengths such as 22, inside tape
 label fields, or in certain types of catalogs. The rule chosen is to cut stuff
 out of the middle, which is better than cutting it purely off either end, but
 eventually somebody makes themselves exactly the right size and shape and
 falls off this cliff.

 --------

 Early IBM VM/CMS command and exec languages tokenize everything to 8
 characters, discarding excesses. On the one hand, this is somewhat less
 hazardous, since it is so pervasive and widespread; but it still causes
 problems. (For one thing, frequent users tend to reduce their entire
 vocabulary to words of 8 letters or less, both a good and a bad effect.... I
 really was doing this for awhile!). The currently preferred script language,
 REXX, avoids this problem, but EXEC I and EXEC II are still in use and still
 do it.

 The problem was especially pernicious in certain circumstances. For example,
 if you wanted to save a token and compare it, you had to concatenate a period
 on the front, because it might be missing, and without at least the period,
 you'd get syntax errors in a comparison. But then, you were limited to 7
 "real" characters, the eighth one having been pushed off the right hand end
 and truncated without notice...

Doug Mosher, 257 Evans, Univ. of California, Berkeley, CA, 415/642-5823 


An ancient computer virus

Joe Dellinger <joe@hanauma.STANFORD.EDU>
Wed, 2 Dec 87 00:14:25 pst
    In 1982, while a student at Texas A+M University, I created a
Virus for Apple ][ Dos 3.3. I was curious to see how long it would take
for such a virus to spread through my own disk collection, so I was very
careful to make the virus completely harmless (and indeed even completely
undetectable). I was very careful to operate under strict quarantine.
Unfortunately, several friends let the virus escape, before I was through
perfecting it. The earlier versions of virus would cause the graphics
in a certain game to smear. Within a few weeks, everybody's (pirated) copy
of this game (called "Congo") stopped working. To correct this situation,
we launched another "perfected" virus to displace the first. As it turned
out, the "perfected" virus worked well, and so I never heard from it again.
    Are Apple ]['s running 64K Dos 3.3 still being used? If so, it might
make an interesting study to see how much this virus has spread in 5 years.
If the virus is in memory, there will be a short ASCII text string listing
the generation count starting at location $B6E8 in memory. Normal DOS has
zeroes there.


Cable violations of privacy (Re: RISKS-5.66)

<rogers%ncrcce%ncrlnk.dayton.ncr.com@RELAY.CS.NET>
Thu, 3 Dec 87 01:34:54 -0500 (at ncrlnk.Dayton.NCR.COM)
People concerned about violations of privacy may be interested in the following
comments from Peter Barton, executive vice president of the Cable Value
Network (CVN - a shop-via-TV retailer) as quoted in the Minneapolis Star
Tribune Sunday magazine:

  "We got your name.  We know where you live.  We know something about what
  your life style may be all about.  We know what you bought, so we're going to
  start sending you catalogs. ...This business is a manifestation of the
  evolution of data transmissions by satellites and the capacity to manage as
  much data as you can.  You couldn't have done this business five years ago.
  These computers weren't powerful enough.  It really is the right business at
  the right time.  It's kind of fun."

Bob Rogers, NCR Comten, St. Paul, MN


Re: Computer-controlled train runs red light (RISKS-5.65)

<ucbcad!ames.UUCP!hoptoad!academ!nuchat!steve@ucbvax.Berkeley.EDU>
2 Dec 87 22:19:44 CST (Wed)
      (Steve Nuchia)

> Technical experts agree that microprocessor-based systems are more flexible
> in operation and much better at monitoring and fault diagnosis than the
> relay-based systems they typically replace. ...

> Symposium participants expressed concern, however, about the probablity
> of failure of the microprocessor in an unsafe way as a result of inadequate
> verification of its software.

Perhaps there is a risk inherent in technological progress - the consumers of
the technology come to expect continued rapid progress, without regard for
engineering reality.

The pair of statements quoted above are an example of such inflated
expectations, and the disappointment that usually accompanies inflated
expectations.  Here we have a bunch of engineers lamenting the lack of
reliability in electronic digital control system, as compared against relay
based controls.  But no mention is made of hardware reliability!

Surely these engineers can't be so paranoid as to think that an exact
duplication of their (primarily digital) relay-based control system in
software would be hard to verify.  It should at least be possible to build a
software implementation that could be easily shown to be equivalent to the
relays, leaving aside the problem of validating an arbitrary "spagetti code"
implementation.

Which brings us to the first excerpt, in which the "chips" are lauded for
being more flexible and having more functionality than the relays.  So, we
are comparing an expensive and mechanically failure-prone solution against a
less expensive solution prone to mysterious bugs and a different breed of
hardware faults.  Trouble is, the two solve different problems!

Why is the computerized solution expected to do everything the relay box did,
plus diagnose itself and be "more flexible in operation" (whatever that means
exactly), at a lower cost and with no technology-specific risks?  At least
these people were responsible and alert enough the realize that they had
expectations that the technology couldn't meet before putting it into
production.

Automobile traffic light control boxes, based on relay technology quite
similar to that used in railroads, fail every so often due to ants building
mounds in the nice warm cabinets.  People have been killed by this bug in a
relay system, yet it fails to generate the kind of emotional response that
software bugs do.  ---

Steve Nuchia  (713) 334 6720


VM systems vulnerability

<Doug Mosher>
Fri, 04 Dec 87 14:15:46 PST
The following has come out in VMSHARE, a national bulletin board for IBM
VM systems programmers. It is in a file "PROB SECURITY", which is only
available to those on a specific prearranged access list.

(Note: the number 2600 derives, I believe, from a frequency that was widely
used in earlier days by blue boxes to permit unpaid long-distance calls.)
                         =============
Append on 12/03/87 at 23:49 by Thomas P. Owens - (907)-276-7600:

     VM has, at long last, arrived!  2600, the Monthly Journal of the American
Hacker, devoted the better part of seven pages (November 1987) to an article
"Hacking IBM's VM/CMS".  The information given is dead(ly) accurate, so far
as it goes.
     I urge one and all to review your anti-hacking measures.  If you don't
have any such measures, this would be a GOOD time to mend your ways.  If your
management is a bit slow to respond to exposures, drop them a broad hint that
there is at this very moment a crew of anti-social computer weenies loose in
your neighborhood.  At worst, the statement is literally true;  at best, a
benevolent "foma".

Doug Mosher, 257 Evans, Univ. of California, Berkeley, CA, 415/642-5823 


Baby monitors end up 'bugging' the whole house

Shane Looker <shane@pepe.cc.umich.edu>
Wed, 2 Dec 87 12:46:26 EST
By : Constance Prater
Original : The Detroit News -- Nov 22, 1987, Section B, page 1
Copied without permission.

When 9-month-old Detroiter Bradley Dunn cries, a lot of people may hear him.
And that "bugs" his mother, Ellen.  But the fact is, Bradley's nursery is
bugged.  Dunn recently put an inexpensive baby monitoring transmitter in
Bradley's room so she could hear him on a receiver she carries throughout
the house.  What she didn't realize was that now any neighbor with a
low-frequency electronic receiver, or walkie-talkie, can eavesdrop on
Bradley -- or any other household activity.

Electronic eavesdropping is a federal [USA] no-no, and upon conviction
under the Electronic Communications Privacy Act of 1986, violators can be
sentenced to a year in prison and fined $100,000.  The law makes it illegal
to intercept, listen to, or disclose information from private electronic
signals.  But federal agents won't be out tracking down neighborhood
listeners, said John Anthony, special agent in Detroit's FBI office.  "From
a practical standpoint, how do you enforce something like that?"  he said.
"We're not going down to investigate baby monitor listeners."

A half-dozen toy companies manufacture the baby monitoring and intercom
devices, including a nationally advertised medel made by Fisher-Price Toys.
The units operate in a frequency range of 30 to 50 megahertz.  Under the
right conditions, they can carry baby gurgles or any other sounds in a room
on low-frequency airwaves to receivers as far away as a quarter of a mile.

Congress probably didn't have baby monitors in mind last year when it wrote
the federal law intended to prevent wiretapping, electronic listening
devices and cabbies stealing each other's fares.  But the nursery listening
aids are, in a sense, "bugs", too.  "You never think that your neighbors
could be listening to you through a baby monitor," said Fran Ganim of
Livonia, who bought a Gerry Deluxe model monitor in August just before the
birth of her daughter.  "I guess we have to watch what we say."

A Taylor man, who owns a Radio Shack PRO 2020 scanner receiver to monitor
police and emergency rescue channels, said he suspected he was breaking the
law when he listened to a husband and wive arguing and yelling at their
children.  "I can pick up everything that's going on in their house," said
the man, who spoke on condition that his name not be used.  He said that
after he also heard a small child crying, he figured out the signal he'd
accidentally locked in on was from a baby monitor.  He traced the signal to
a home a half-block away and eventually told his neighbors about their lack
of privacy.

Baby monitors, beepers, pagers, office intercoms, cordless and car
telephones and walkie-talkies all emit electronic signals.  The devices can
transmit or receive signals from each other in close range, or be picked up
on more powerful receivers capable of scanning lower frequencies.

Electronics technology has improved faster than legislatures can make laws
regulating the equipment's use, the FBI's Anthony said.  "Anybody with a
little bit of knowledge and a lot of time on their hands ... can go to Radio
Shack or another electronics store, get some pretty ophisticated equipment
and listen to just about anything they want," he said.  "When you throw it
(a transmitting signal) up there in the ionosphere, anybody can hear you."

The Federal Communications Commission (FCC) requires label warnings only for
users of cordless telephones, which use shared frequencies, said Richard
Engleman, and FCC electronics engineer.  "Communications are intended to be
private.  I would be upset if somebody was listening to me," Engleman said.
He said electronic interception is a difficult area to regulate, citing
satellite dishes as an example.  "A lot of people maintain the airwaves are
free to use.  And what comes into my home is fair game," he said.


F4 in 'Nam (Re: Reversed signal polarity causing accidents)

Brent Chapman <chapman%mica.Berkeley.EDU@violet.berkeley.edu>
Mon, 30 Nov 87 23:06:07 PST
In Risks-5.66, Clive D.W. Feather <mcvax!root.co.uk!cdwf@uunet.uu.net> writes
of how reversed polarity in a rail switching relay led to an accident that
killed two people.

This reminds me of story I heard about the American F4 fighter in its
early days of service in VietNam.  I heard the story from an Air Force
colonel who flew that aircraft in VietNam; I don't know if he was being
completely honest with me or not, but it seems plausible. 

During an ejection, the charges that separate the canopy are supposed to
explode a split second before the charges that propell the ejection seat
out of the aircraft.  In these early F4's, there was apparently about a
50-50 chance that the sequence would happen the other way around (the
seat would fire before the canopy did), causing the pilot to be ejected
through the still-intact glass canopy, usually seriously or fatally
injuring the pilot.  The culprit was a wiring harness plug in the
ejection control system that could be connected (with proper and common
application of main force) so that the "blow-canopy" and "blow-seat"
lines were reversed, causing the order of firing to be reversed.  When
the problem was finally diagnosed, the plugs were all replaced with ones
that supposedly _couldn't_ be reversed. 

An interesting side note: the low-tech means that the pilots developed
to deal with the problem between the time they decided it _was_ a
problem and the somewhat later time that it was "fixed" was to wire a
pair of bayonets to the "rails" on either side of the ejection seat so
that the points projected above the pilot's head.  That way, if the seat
blew before the canopy, the canopy would be shattered by the knives
instead of by the pilot's head. 

Brent Chapman        chapman@mica.berkeley.edu or ucbvax!mica!chapman


IRS computers (yet again!)

Joe Morris (jcmorris@mitre.arpa) <jcmorris@mitre.arpa>
Wed, 02 Dec 87 08:50:20 EST
The following short item appeared on  p. 1 of the 25 November issue of
_The_Wall_Street_Journal_; it's copied in its entirity, and as usual
without permission:

    DON'T BLAME US.  It's our naughty computer that keeps breaking
  the law.

    The law tells the IRS to wait 90 days after issuing a deficiency
  notice before trying to collect.  If a taxpayer takes the matter to 
  Tax Court, the IRS must hold off until the case is resolved.  Yet,
  soon after Paul and Gina Husby proceeded to Tax Court, the IRS billed
  the San Francisco couple $47,000.  A computer error, an IRS attorney 
  assured them; don't worry.

    But more bills came, and the agency grabbed nearly $3,800 in Paul's
  credit union.  Even after a federal court ordered a halt, the unlawful
  collection action went on.  Finally, the IRS curbed itself, returned
  the credit union money, but argued the Husby's lawsuit for damages
  should be summarily dismissed.  District Court Judge Weigel recently
  noted that the IRS "position boils down to the contention that the 
  whole unfortunate incident was really nobody's fault, but the
  computers'."

    That won't let the agency off the hook, the judge said, ruling that 
  the case should proceed to trial.


Call for papers -- JOURNAL OF COMPUTING AND SOCIETY

Gary Chapman <chapman@russell.stanford.edu>
Thu, 3 Dec 87 14:35:53 PST
                                CALL FOR PAPERS
                                FOR DEBUT ISSUE
                                      of
                     THE JOURNAL OF COMPUTING AND SOCIETY

Ablex Publishing Corporation will begin quarterly publication of a new academic
journal on the social implications of computing technology in late 1988 or
early 1989.  The purpose of The Journal of Computing and Society will be to
stimulate lively debate and speculation on a wide variety of topics concerning
the computerization of society.  The journal will be refereed.  When it begins
publication, individual subscriptions will be encouraged, and it should be
available on stands in quality bookstores.

Issues of the journal will be organized around themes.  The first theme is "Has
There Been A Computer Revolution?"  Contributors are encouraged to submit
papers relating to this theme (not necessarily answering the question yea or
nay).  Papers for this issue should be received by April 15, 1988.  Future
themes will include computers and war, computers and privacy, risks to the
public, computers and power, the computer as metaphor, computers and gender,
etc.  Suggestions for themes are welcome, and the journal will publish a few
articles outside the theme if the papers are of signficant quality, timeliness,
and relevance to the purposes of the journal.

Manuscripts should be 10-20 pages in length, conforming to the Chicago Manual
of Style, and submitted in quadruplicate.  A sheet of information for authors
is available from the editor.

The editor of this journal is Gary Chapman, executive director of Computer
Professionals for Social Responsibility.  Manuscript submissions and all
correspondence should be directed to him at P.O. Box 717, Palo Alto, CA 94301.
The telephone number is (415) 322-3778.

The current editorial board of The Journal of Computing and Society consists of
the following people:

Jerry Berman                       Rob Kling             Luca Simoncini
Margaret Boden                     John Ladd             Brian Smith
David Burnham                      Abbe Mowshowitz       Lucy Suchman
Hubert Dreyfus                     Peter Neumann         C.S. Tang
Jean-Louis Gassee                  Susan H. Nycum        Joseph Weizenbaum
Calvin Gotlieb                     Kristen Nygaard       Alan F. Westin
Douglas Hofstadter                 Paul Saffo            Langdon Winner
Deborah Johnson                    Mike Sharples         Terry Winograd
                                   Lenny Siegel

Please report problems with the web pages to the maintainer

Top