Please try the URL privacy information feature enabled by clicking the flashlight icon above. This will reveal two icons after each link the body of the digest. The shield takes you to a breakdown of Terms of Service for the site - however only a small number of sites are covered at the moment. The flashlight take you to an analysis of the various trackers etc. that the linked site delivers. Please let the website maintainer know if you find this useful or not. As a RISKS reader, you will probably not be surprised by what is revealed…
I am interested in the legal aspects of using expert systems.
Consider, and please comment on, this scenario.
* * * * * * * * * * *
A well-respected, well-established expert systems(ES) company constructs
an expert financial advisory system. The firm employs the top ES
applications specialists in the country. The system is constructed with
help from the top domain experts in the financial services industry. It
is exhaustively tested, including verification of rules, verification of
reasoning, and further analyses to establish the system's overall value.
All results are excellent, and the system is offered for sale.
Joe Smith is looking for a financial advisory system. He reads the sales
literature, which lists names of experts whose advice was used when
building the system. It lists the credentials of the people in the
company who were the implementors. It lists names of satisfied users,
and quotes comments that praise the product. Joe wavers, weakens, and
buys the product.
"The product IS good,", Joe explains. "I got it up and running in less
than an hour!" Joe spends the remainder of that evening entering his own
personal financial data, answering questions asked by the ES, and
anticipating the results.
By now, you know the outcome. On the Friday morning before Black Monday,
the expert system tells Joe to "sell everything he has and go into the
stock market." ESs can usually explain their actions, and Joe asks for
an explanation. The ES replies "because ... it's only been going UP for
the past five years and there are NO PROBLEMS IN SIGHT."
Joe loses big on Monday. Since he lives in California, (where there is
one lawyer for every four households, or so it seems, and a motion
asking that a lawsuit be declared frivolous is itself declared
frivolous) he is going to sue someone. But who?
The company that implemented the system?
The domain experts that built their advice into the system?
The knowledge engineers who turned expertise into a system?
The distributor who sold an obviously defective product?
Will a warranty protect the parties involved? Probably not. If real
damages are involved, people will file lawsuits anyway.
Can the domain experts hide behind the company? Probably not. The
company will specifically want to use their names and reputations as the
source of credibility for the product. The user's reaction could be,
"There's the so-and-so who told me to go into the stock market."
Can the knowledge engineers be sued for faulty construction of a system?
Why not, when people who build anything else badly can be sued?
How about the distributor — after all, he ultimately took money from
the customer and gave him the product.
* * * * * * * * * * *
I would be very interested in any of your thoughts on this subject. I'd
be happy to summarize the responses to the net.
Barry A. Stevens, Applied AI Systems, Inc., PO Box 2747, Del Mar, CA 92014
619-755-7231
Northwest Airlink had a real computer crash. A 50-pound personal computer belonging to a passenger, Ron Olstad, fell from a cargo pod on a Jet Stream 31 commuter plane landing in Oshkosh, Wisconsin. The computer crash landed into Ronald Miller's backyard. Northwest Airlink replaced the computer. [but not the divot!] From the International Herald Tribune, 18 November 1987, p.3, courtesy of Vicki Almstrum of Philips in Jarfalla (with `"' over the first two `a's).
BLUE CROSS ISSUES RED-FACED APOLOGY (Bergen Record, 12/2/87) UPI, Providence, R.I.- A mysterious prankster has struck Blue Cross and Blue Shield of Rhode Island and about 100 of its subscribers. The subscribers recently received letters from the health insurer concerning doctor bills, with a postscript that would confirm the worst fears of many subscribers. It read, "Note: Your eligible charges will remain in file until hell freezes over." Red-faced Blue Cross officials apologized to subscribers for the blooper but have not been able to find the culprit. A temporary employee who had access to the computer that generated the letter left the day the sentence was added, a spokesman said Monday.
Following the discussions of truncating UNIX keywords to 8 characters without
notice, I would like to make the following statement/opinion/recommendation:
It is ALWAYS BAD PRACTICE to delete anything without notice.
One special case of this is:
It is ALWAYS BAD PRACTICE to truncate anything without notice.
Many examples over the years occur to me; here's a small partial list.
--------
Several products provide a really gimpy form of "Artificial Intelligence"
by allowing the user to insert meaningless words, and ignoring
them without notice.
Example: syntax would be: "print amount by state by city".
Syntax also allowed: "please print me a report showing the
amount sorted by state and then by city".
This starts off looking good, but ends up deleting all sorts of necessary
keywords if you ever slip and misspell them, with very hazardous outcomes. You
get a report, all right; no messages are issued; you go ahead and buy
things/fire people/explode bombs, based on the WRONG REPORT RESULTS.
Products which do this include FOCUS and TELL-A-GRAF.
--------
Some Microsoft BASIC interpreters give the impression of allowing variable
names longer than 2 characters, but actually they just truncate to two
characters. This results in VERY PAINFUL and hard to find bugs, when one
finally calls two things PRINT and PRACTICE or whatever.
--------
The IBM Pl/I compiler truncates variable names to 31 characters. This is
longer than most people are motivated to use so it has rarely caused anyone
pain. External names are truncated to 8 characters, but at least notice is
given.
--------
IBM MVS dataset names are truncated at 44 characters. This is not a frequent
source of problems, but many folks have tripped over this at one time or
another. To compound things, under various circumstances in the MVS operating
system, dataset names are further shrunk to lengths such as 22, inside tape
label fields, or in certain types of catalogs. The rule chosen is to cut stuff
out of the middle, which is better than cutting it purely off either end, but
eventually somebody makes themselves exactly the right size and shape and
falls off this cliff.
--------
Early IBM VM/CMS command and exec languages tokenize everything to 8
characters, discarding excesses. On the one hand, this is somewhat less
hazardous, since it is so pervasive and widespread; but it still causes
problems. (For one thing, frequent users tend to reduce their entire
vocabulary to words of 8 letters or less, both a good and a bad effect.... I
really was doing this for awhile!). The currently preferred script language,
REXX, avoids this problem, but EXEC I and EXEC II are still in use and still
do it.
The problem was especially pernicious in certain circumstances. For example,
if you wanted to save a token and compare it, you had to concatenate a period
on the front, because it might be missing, and without at least the period,
you'd get syntax errors in a comparison. But then, you were limited to 7
"real" characters, the eighth one having been pushed off the right hand end
and truncated without notice...
Doug Mosher, 257 Evans, Univ. of California, Berkeley, CA, 415/642-5823
In 1982, while a student at Texas A+M University, I created a
Virus for Apple ][ Dos 3.3. I was curious to see how long it would take
for such a virus to spread through my own disk collection, so I was very
careful to make the virus completely harmless (and indeed even completely
undetectable). I was very careful to operate under strict quarantine.
Unfortunately, several friends let the virus escape, before I was through
perfecting it. The earlier versions of virus would cause the graphics
in a certain game to smear. Within a few weeks, everybody's (pirated) copy
of this game (called "Congo") stopped working. To correct this situation,
we launched another "perfected" virus to displace the first. As it turned
out, the "perfected" virus worked well, and so I never heard from it again.
Are Apple ]['s running 64K Dos 3.3 still being used? If so, it might
make an interesting study to see how much this virus has spread in 5 years.
If the virus is in memory, there will be a short ASCII text string listing
the generation count starting at location $B6E8 in memory. Normal DOS has
zeroes there.
People concerned about violations of privacy may be interested in the following comments from Peter Barton, executive vice president of the Cable Value Network (CVN - a shop-via-TV retailer) as quoted in the Minneapolis Star Tribune Sunday magazine: "We got your name. We know where you live. We know something about what your life style may be all about. We know what you bought, so we're going to start sending you catalogs. ...This business is a manifestation of the evolution of data transmissions by satellites and the capacity to manage as much data as you can. You couldn't have done this business five years ago. These computers weren't powerful enough. It really is the right business at the right time. It's kind of fun." Bob Rogers, NCR Comten, St. Paul, MN
(Steve Nuchia)
> Technical experts agree that microprocessor-based systems are more flexible
> in operation and much better at monitoring and fault diagnosis than the
> relay-based systems they typically replace. ...
> Symposium participants expressed concern, however, about the probablity
> of failure of the microprocessor in an unsafe way as a result of inadequate
> verification of its software.
Perhaps there is a risk inherent in technological progress - the consumers of
the technology come to expect continued rapid progress, without regard for
engineering reality.
The pair of statements quoted above are an example of such inflated
expectations, and the disappointment that usually accompanies inflated
expectations. Here we have a bunch of engineers lamenting the lack of
reliability in electronic digital control system, as compared against relay
based controls. But no mention is made of hardware reliability!
Surely these engineers can't be so paranoid as to think that an exact
duplication of their (primarily digital) relay-based control system in
software would be hard to verify. It should at least be possible to build a
software implementation that could be easily shown to be equivalent to the
relays, leaving aside the problem of validating an arbitrary "spagetti code"
implementation.
Which brings us to the first excerpt, in which the "chips" are lauded for
being more flexible and having more functionality than the relays. So, we
are comparing an expensive and mechanically failure-prone solution against a
less expensive solution prone to mysterious bugs and a different breed of
hardware faults. Trouble is, the two solve different problems!
Why is the computerized solution expected to do everything the relay box did,
plus diagnose itself and be "more flexible in operation" (whatever that means
exactly), at a lower cost and with no technology-specific risks? At least
these people were responsible and alert enough the realize that they had
expectations that the technology couldn't meet before putting it into
production.
Automobile traffic light control boxes, based on relay technology quite
similar to that used in railroads, fail every so often due to ants building
mounds in the nice warm cabinets. People have been killed by this bug in a
relay system, yet it fails to generate the kind of emotional response that
software bugs do. ---
Steve Nuchia (713) 334 6720
The following has come out in VMSHARE, a national bulletin board for IBM
VM systems programmers. It is in a file "PROB SECURITY", which is only
available to those on a specific prearranged access list.
(Note: the number 2600 derives, I believe, from a frequency that was widely
used in earlier days by blue boxes to permit unpaid long-distance calls.)
=============
Append on 12/03/87 at 23:49 by Thomas P. Owens - (907)-276-7600:
VM has, at long last, arrived! 2600, the Monthly Journal of the American
Hacker, devoted the better part of seven pages (November 1987) to an article
"Hacking IBM's VM/CMS". The information given is dead(ly) accurate, so far
as it goes.
I urge one and all to review your anti-hacking measures. If you don't
have any such measures, this would be a GOOD time to mend your ways. If your
management is a bit slow to respond to exposures, drop them a broad hint that
there is at this very moment a crew of anti-social computer weenies loose in
your neighborhood. At worst, the statement is literally true; at best, a
benevolent "foma".
Doug Mosher, 257 Evans, Univ. of California, Berkeley, CA, 415/642-5823
By : Constance Prater Original : The Detroit News — Nov 22, 1987, Section B, page 1 Copied without permission. When 9-month-old Detroiter Bradley Dunn cries, a lot of people may hear him. And that "bugs" his mother, Ellen. But the fact is, Bradley's nursery is bugged. Dunn recently put an inexpensive baby monitoring transmitter in Bradley's room so she could hear him on a receiver she carries throughout the house. What she didn't realize was that now any neighbor with a low-frequency electronic receiver, or walkie-talkie, can eavesdrop on Bradley — or any other household activity. Electronic eavesdropping is a federal [USA] no-no, and upon conviction under the Electronic Communications Privacy Act of 1986, violators can be sentenced to a year in prison and fined $100,000. The law makes it illegal to intercept, listen to, or disclose information from private electronic signals. But federal agents won't be out tracking down neighborhood listeners, said John Anthony, special agent in Detroit's FBI office. "From a practical standpoint, how do you enforce something like that?" he said. "We're not going down to investigate baby monitor listeners." A half-dozen toy companies manufacture the baby monitoring and intercom devices, including a nationally advertised medel made by Fisher-Price Toys. The units operate in a frequency range of 30 to 50 megahertz. Under the right conditions, they can carry baby gurgles or any other sounds in a room on low-frequency airwaves to receivers as far away as a quarter of a mile. Congress probably didn't have baby monitors in mind last year when it wrote the federal law intended to prevent wiretapping, electronic listening devices and cabbies stealing each other's fares. But the nursery listening aids are, in a sense, "bugs", too. "You never think that your neighbors could be listening to you through a baby monitor," said Fran Ganim of Livonia, who bought a Gerry Deluxe model monitor in August just before the birth of her daughter. "I guess we have to watch what we say." A Taylor man, who owns a Radio Shack PRO 2020 scanner receiver to monitor police and emergency rescue channels, said he suspected he was breaking the law when he listened to a husband and wive arguing and yelling at their children. "I can pick up everything that's going on in their house," said the man, who spoke on condition that his name not be used. He said that after he also heard a small child crying, he figured out the signal he'd accidentally locked in on was from a baby monitor. He traced the signal to a home a half-block away and eventually told his neighbors about their lack of privacy. Baby monitors, beepers, pagers, office intercoms, cordless and car telephones and walkie-talkies all emit electronic signals. The devices can transmit or receive signals from each other in close range, or be picked up on more powerful receivers capable of scanning lower frequencies. Electronics technology has improved faster than legislatures can make laws regulating the equipment's use, the FBI's Anthony said. "Anybody with a little bit of knowledge and a lot of time on their hands ... can go to Radio Shack or another electronics store, get some pretty ophisticated equipment and listen to just about anything they want," he said. "When you throw it (a transmitting signal) up there in the ionosphere, anybody can hear you." The Federal Communications Commission (FCC) requires label warnings only for users of cordless telephones, which use shared frequencies, said Richard Engleman, and FCC electronics engineer. "Communications are intended to be private. I would be upset if somebody was listening to me," Engleman said. He said electronic interception is a difficult area to regulate, citing satellite dishes as an example. "A lot of people maintain the airwaves are free to use. And what comes into my home is fair game," he said.
In Risks-5.66, Clive D.W. Feather <mcvax!root.co.uk!cdwf@uunet.uu.net> writes of how reversed polarity in a rail switching relay led to an accident that killed two people. This reminds me of story I heard about the American F4 fighter in its early days of service in VietNam. I heard the story from an Air Force colonel who flew that aircraft in VietNam; I don't know if he was being completely honest with me or not, but it seems plausible. During an ejection, the charges that separate the canopy are supposed to explode a split second before the charges that propell the ejection seat out of the aircraft. In these early F4's, there was apparently about a 50-50 chance that the sequence would happen the other way around (the seat would fire before the canopy did), causing the pilot to be ejected through the still-intact glass canopy, usually seriously or fatally injuring the pilot. The culprit was a wiring harness plug in the ejection control system that could be connected (with proper and common application of main force) so that the "blow-canopy" and "blow-seat" lines were reversed, causing the order of firing to be reversed. When the problem was finally diagnosed, the plugs were all replaced with ones that supposedly _couldn't_ be reversed. An interesting side note: the low-tech means that the pilots developed to deal with the problem between the time they decided it _was_ a problem and the somewhat later time that it was "fixed" was to wire a pair of bayonets to the "rails" on either side of the ejection seat so that the points projected above the pilot's head. That way, if the seat blew before the canopy, the canopy would be shattered by the knives instead of by the pilot's head. Brent Chapman chapman@mica.berkeley.edu or ucbvax!mica!chapman
The following short item appeared on p. 1 of the 25 November issue of
_The_Wall_Street_Journal_; it's copied in its entirity, and as usual
without permission:
DON'T BLAME US. It's our naughty computer that keeps breaking
the law.
The law tells the IRS to wait 90 days after issuing a deficiency
notice before trying to collect. If a taxpayer takes the matter to
Tax Court, the IRS must hold off until the case is resolved. Yet,
soon after Paul and Gina Husby proceeded to Tax Court, the IRS billed
the San Francisco couple $47,000. A computer error, an IRS attorney
assured them; don't worry.
But more bills came, and the agency grabbed nearly $3,800 in Paul's
credit union. Even after a federal court ordered a halt, the unlawful
collection action went on. Finally, the IRS curbed itself, returned
the credit union money, but argued the Husby's lawsuit for damages
should be summarily dismissed. District Court Judge Weigel recently
noted that the IRS "position boils down to the contention that the
whole unfortunate incident was really nobody's fault, but the
computers'."
That won't let the agency off the hook, the judge said, ruling that
the case should proceed to trial.
CALL FOR PAPERS
FOR DEBUT ISSUE
of
THE JOURNAL OF COMPUTING AND SOCIETY
Ablex Publishing Corporation will begin quarterly publication of a new academic
journal on the social implications of computing technology in late 1988 or
early 1989. The purpose of The Journal of Computing and Society will be to
stimulate lively debate and speculation on a wide variety of topics concerning
the computerization of society. The journal will be refereed. When it begins
publication, individual subscriptions will be encouraged, and it should be
available on stands in quality bookstores.
Issues of the journal will be organized around themes. The first theme is "Has
There Been A Computer Revolution?" Contributors are encouraged to submit
papers relating to this theme (not necessarily answering the question yea or
nay). Papers for this issue should be received by April 15, 1988. Future
themes will include computers and war, computers and privacy, risks to the
public, computers and power, the computer as metaphor, computers and gender,
etc. Suggestions for themes are welcome, and the journal will publish a few
articles outside the theme if the papers are of signficant quality, timeliness,
and relevance to the purposes of the journal.
Manuscripts should be 10-20 pages in length, conforming to the Chicago Manual
of Style, and submitted in quadruplicate. A sheet of information for authors
is available from the editor.
The editor of this journal is Gary Chapman, executive director of Computer
Professionals for Social Responsibility. Manuscript submissions and all
correspondence should be directed to him at P.O. Box 717, Palo Alto, CA 94301.
The telephone number is (415) 322-3778.
The current editorial board of The Journal of Computing and Society consists of
the following people:
Jerry Berman Rob Kling Luca Simoncini
Margaret Boden John Ladd Brian Smith
David Burnham Abbe Mowshowitz Lucy Suchman
Hubert Dreyfus Peter Neumann C.S. Tang
Jean-Louis Gassee Susan H. Nycum Joseph Weizenbaum
Calvin Gotlieb Kristen Nygaard Alan F. Westin
Douglas Hofstadter Paul Saffo Langdon Winner
Deborah Johnson Mike Sharples Terry Winograd
Lenny Siegel
Please report problems with the web pages to the maintainer