I am interested in the legal aspects of using expert systems. Consider, and please comment on, this scenario. * * * * * * * * * * * A well-respected, well-established expert systems(ES) company constructs an expert financial advisory system. The firm employs the top ES applications specialists in the country. The system is constructed with help from the top domain experts in the financial services industry. It is exhaustively tested, including verification of rules, verification of reasoning, and further analyses to establish the system's overall value. All results are excellent, and the system is offered for sale. Joe Smith is looking for a financial advisory system. He reads the sales literature, which lists names of experts whose advice was used when building the system. It lists the credentials of the people in the company who were the implementors. It lists names of satisfied users, and quotes comments that praise the product. Joe wavers, weakens, and buys the product. "The product IS good,", Joe explains. "I got it up and running in less than an hour!" Joe spends the remainder of that evening entering his own personal financial data, answering questions asked by the ES, and anticipating the results. By now, you know the outcome. On the Friday morning before Black Monday, the expert system tells Joe to "sell everything he has and go into the stock market." ESs can usually explain their actions, and Joe asks for an explanation. The ES replies "because ... it's only been going UP for the past five years and there are NO PROBLEMS IN SIGHT." Joe loses big on Monday. Since he lives in California, (where there is one lawyer for every four households, or so it seems, and a motion asking that a lawsuit be declared frivolous is itself declared frivolous) he is going to sue someone. But who? The company that implemented the system? The domain experts that built their advice into the system? The knowledge engineers who turned expertise into a system? The distributor who sold an obviously defective product? Will a warranty protect the parties involved? Probably not. If real damages are involved, people will file lawsuits anyway. Can the domain experts hide behind the company? Probably not. The company will specifically want to use their names and reputations as the source of credibility for the product. The user's reaction could be, "There's the so-and-so who told me to go into the stock market." Can the knowledge engineers be sued for faulty construction of a system? Why not, when people who build anything else badly can be sued? How about the distributor — after all, he ultimately took money from the customer and gave him the product. * * * * * * * * * * * I would be very interested in any of your thoughts on this subject. I'd be happy to summarize the responses to the net. Barry A. Stevens, Applied AI Systems, Inc., PO Box 2747, Del Mar, CA 92014 619-755-7231
Northwest Airlink had a real computer crash. A 50-pound personal computer belonging to a passenger, Ron Olstad, fell from a cargo pod on a Jet Stream 31 commuter plane landing in Oshkosh, Wisconsin. The computer crash landed into Ronald Miller's backyard. Northwest Airlink replaced the computer. [but not the divot!] From the International Herald Tribune, 18 November 1987, p.3, courtesy of Vicki Almstrum of Philips in Jarfalla (with `"' over the first two `a's).
BLUE CROSS ISSUES RED-FACED APOLOGY (Bergen Record, 12/2/87) UPI, Providence, R.I.- A mysterious prankster has struck Blue Cross and Blue Shield of Rhode Island and about 100 of its subscribers. The subscribers recently received letters from the health insurer concerning doctor bills, with a postscript that would confirm the worst fears of many subscribers. It read, "Note: Your eligible charges will remain in file until hell freezes over." Red-faced Blue Cross officials apologized to subscribers for the blooper but have not been able to find the culprit. A temporary employee who had access to the computer that generated the letter left the day the sentence was added, a spokesman said Monday.
Following the discussions of truncating UNIX keywords to 8 characters without notice, I would like to make the following statement/opinion/recommendation: It is ALWAYS BAD PRACTICE to delete anything without notice. One special case of this is: It is ALWAYS BAD PRACTICE to truncate anything without notice. Many examples over the years occur to me; here's a small partial list. -------- Several products provide a really gimpy form of "Artificial Intelligence" by allowing the user to insert meaningless words, and ignoring them without notice. Example: syntax would be: "print amount by state by city". Syntax also allowed: "please print me a report showing the amount sorted by state and then by city". This starts off looking good, but ends up deleting all sorts of necessary keywords if you ever slip and misspell them, with very hazardous outcomes. You get a report, all right; no messages are issued; you go ahead and buy things/fire people/explode bombs, based on the WRONG REPORT RESULTS. Products which do this include FOCUS and TELL-A-GRAF. -------- Some Microsoft BASIC interpreters give the impression of allowing variable names longer than 2 characters, but actually they just truncate to two characters. This results in VERY PAINFUL and hard to find bugs, when one finally calls two things PRINT and PRACTICE or whatever. -------- The IBM Pl/I compiler truncates variable names to 31 characters. This is longer than most people are motivated to use so it has rarely caused anyone pain. External names are truncated to 8 characters, but at least notice is given. -------- IBM MVS dataset names are truncated at 44 characters. This is not a frequent source of problems, but many folks have tripped over this at one time or another. To compound things, under various circumstances in the MVS operating system, dataset names are further shrunk to lengths such as 22, inside tape label fields, or in certain types of catalogs. The rule chosen is to cut stuff out of the middle, which is better than cutting it purely off either end, but eventually somebody makes themselves exactly the right size and shape and falls off this cliff. -------- Early IBM VM/CMS command and exec languages tokenize everything to 8 characters, discarding excesses. On the one hand, this is somewhat less hazardous, since it is so pervasive and widespread; but it still causes problems. (For one thing, frequent users tend to reduce their entire vocabulary to words of 8 letters or less, both a good and a bad effect.... I really was doing this for awhile!). The currently preferred script language, REXX, avoids this problem, but EXEC I and EXEC II are still in use and still do it. The problem was especially pernicious in certain circumstances. For example, if you wanted to save a token and compare it, you had to concatenate a period on the front, because it might be missing, and without at least the period, you'd get syntax errors in a comparison. But then, you were limited to 7 "real" characters, the eighth one having been pushed off the right hand end and truncated without notice... Doug Mosher, 257 Evans, Univ. of California, Berkeley, CA, 415/642-5823
In 1982, while a student at Texas A+M University, I created a Virus for Apple ][ Dos 3.3. I was curious to see how long it would take for such a virus to spread through my own disk collection, so I was very careful to make the virus completely harmless (and indeed even completely undetectable). I was very careful to operate under strict quarantine. Unfortunately, several friends let the virus escape, before I was through perfecting it. The earlier versions of virus would cause the graphics in a certain game to smear. Within a few weeks, everybody's (pirated) copy of this game (called "Congo") stopped working. To correct this situation, we launched another "perfected" virus to displace the first. As it turned out, the "perfected" virus worked well, and so I never heard from it again. Are Apple ]['s running 64K Dos 3.3 still being used? If so, it might make an interesting study to see how much this virus has spread in 5 years. If the virus is in memory, there will be a short ASCII text string listing the generation count starting at location $B6E8 in memory. Normal DOS has zeroes there.
People concerned about violations of privacy may be interested in the following comments from Peter Barton, executive vice president of the Cable Value Network (CVN - a shop-via-TV retailer) as quoted in the Minneapolis Star Tribune Sunday magazine: "We got your name. We know where you live. We know something about what your life style may be all about. We know what you bought, so we're going to start sending you catalogs. ...This business is a manifestation of the evolution of data transmissions by satellites and the capacity to manage as much data as you can. You couldn't have done this business five years ago. These computers weren't powerful enough. It really is the right business at the right time. It's kind of fun." Bob Rogers, NCR Comten, St. Paul, MN
(Steve Nuchia) > Technical experts agree that microprocessor-based systems are more flexible > in operation and much better at monitoring and fault diagnosis than the > relay-based systems they typically replace. ... > Symposium participants expressed concern, however, about the probablity > of failure of the microprocessor in an unsafe way as a result of inadequate > verification of its software. Perhaps there is a risk inherent in technological progress - the consumers of the technology come to expect continued rapid progress, without regard for engineering reality. The pair of statements quoted above are an example of such inflated expectations, and the disappointment that usually accompanies inflated expectations. Here we have a bunch of engineers lamenting the lack of reliability in electronic digital control system, as compared against relay based controls. But no mention is made of hardware reliability! Surely these engineers can't be so paranoid as to think that an exact duplication of their (primarily digital) relay-based control system in software would be hard to verify. It should at least be possible to build a software implementation that could be easily shown to be equivalent to the relays, leaving aside the problem of validating an arbitrary "spagetti code" implementation. Which brings us to the first excerpt, in which the "chips" are lauded for being more flexible and having more functionality than the relays. So, we are comparing an expensive and mechanically failure-prone solution against a less expensive solution prone to mysterious bugs and a different breed of hardware faults. Trouble is, the two solve different problems! Why is the computerized solution expected to do everything the relay box did, plus diagnose itself and be "more flexible in operation" (whatever that means exactly), at a lower cost and with no technology-specific risks? At least these people were responsible and alert enough the realize that they had expectations that the technology couldn't meet before putting it into production. Automobile traffic light control boxes, based on relay technology quite similar to that used in railroads, fail every so often due to ants building mounds in the nice warm cabinets. People have been killed by this bug in a relay system, yet it fails to generate the kind of emotional response that software bugs do. --- Steve Nuchia (713) 334 6720
The following has come out in VMSHARE, a national bulletin board for IBM VM systems programmers. It is in a file "PROB SECURITY", which is only available to those on a specific prearranged access list. (Note: the number 2600 derives, I believe, from a frequency that was widely used in earlier days by blue boxes to permit unpaid long-distance calls.) ============= Append on 12/03/87 at 23:49 by Thomas P. Owens - (907)-276-7600: VM has, at long last, arrived! 2600, the Monthly Journal of the American Hacker, devoted the better part of seven pages (November 1987) to an article "Hacking IBM's VM/CMS". The information given is dead(ly) accurate, so far as it goes. I urge one and all to review your anti-hacking measures. If you don't have any such measures, this would be a GOOD time to mend your ways. If your management is a bit slow to respond to exposures, drop them a broad hint that there is at this very moment a crew of anti-social computer weenies loose in your neighborhood. At worst, the statement is literally true; at best, a benevolent "foma". Doug Mosher, 257 Evans, Univ. of California, Berkeley, CA, 415/642-5823
By : Constance Prater Original : The Detroit News — Nov 22, 1987, Section B, page 1 Copied without permission. When 9-month-old Detroiter Bradley Dunn cries, a lot of people may hear him. And that "bugs" his mother, Ellen. But the fact is, Bradley's nursery is bugged. Dunn recently put an inexpensive baby monitoring transmitter in Bradley's room so she could hear him on a receiver she carries throughout the house. What she didn't realize was that now any neighbor with a low-frequency electronic receiver, or walkie-talkie, can eavesdrop on Bradley — or any other household activity. Electronic eavesdropping is a federal [USA] no-no, and upon conviction under the Electronic Communications Privacy Act of 1986, violators can be sentenced to a year in prison and fined $100,000. The law makes it illegal to intercept, listen to, or disclose information from private electronic signals. But federal agents won't be out tracking down neighborhood listeners, said John Anthony, special agent in Detroit's FBI office. "From a practical standpoint, how do you enforce something like that?" he said. "We're not going down to investigate baby monitor listeners." A half-dozen toy companies manufacture the baby monitoring and intercom devices, including a nationally advertised medel made by Fisher-Price Toys. The units operate in a frequency range of 30 to 50 megahertz. Under the right conditions, they can carry baby gurgles or any other sounds in a room on low-frequency airwaves to receivers as far away as a quarter of a mile. Congress probably didn't have baby monitors in mind last year when it wrote the federal law intended to prevent wiretapping, electronic listening devices and cabbies stealing each other's fares. But the nursery listening aids are, in a sense, "bugs", too. "You never think that your neighbors could be listening to you through a baby monitor," said Fran Ganim of Livonia, who bought a Gerry Deluxe model monitor in August just before the birth of her daughter. "I guess we have to watch what we say." A Taylor man, who owns a Radio Shack PRO 2020 scanner receiver to monitor police and emergency rescue channels, said he suspected he was breaking the law when he listened to a husband and wive arguing and yelling at their children. "I can pick up everything that's going on in their house," said the man, who spoke on condition that his name not be used. He said that after he also heard a small child crying, he figured out the signal he'd accidentally locked in on was from a baby monitor. He traced the signal to a home a half-block away and eventually told his neighbors about their lack of privacy. Baby monitors, beepers, pagers, office intercoms, cordless and car telephones and walkie-talkies all emit electronic signals. The devices can transmit or receive signals from each other in close range, or be picked up on more powerful receivers capable of scanning lower frequencies. Electronics technology has improved faster than legislatures can make laws regulating the equipment's use, the FBI's Anthony said. "Anybody with a little bit of knowledge and a lot of time on their hands ... can go to Radio Shack or another electronics store, get some pretty ophisticated equipment and listen to just about anything they want," he said. "When you throw it (a transmitting signal) up there in the ionosphere, anybody can hear you." The Federal Communications Commission (FCC) requires label warnings only for users of cordless telephones, which use shared frequencies, said Richard Engleman, and FCC electronics engineer. "Communications are intended to be private. I would be upset if somebody was listening to me," Engleman said. He said electronic interception is a difficult area to regulate, citing satellite dishes as an example. "A lot of people maintain the airwaves are free to use. And what comes into my home is fair game," he said.
In Risks-5.66, Clive D.W. Feather <firstname.lastname@example.org> writes of how reversed polarity in a rail switching relay led to an accident that killed two people. This reminds me of story I heard about the American F4 fighter in its early days of service in VietNam. I heard the story from an Air Force colonel who flew that aircraft in VietNam; I don't know if he was being completely honest with me or not, but it seems plausible. During an ejection, the charges that separate the canopy are supposed to explode a split second before the charges that propell the ejection seat out of the aircraft. In these early F4's, there was apparently about a 50-50 chance that the sequence would happen the other way around (the seat would fire before the canopy did), causing the pilot to be ejected through the still-intact glass canopy, usually seriously or fatally injuring the pilot. The culprit was a wiring harness plug in the ejection control system that could be connected (with proper and common application of main force) so that the "blow-canopy" and "blow-seat" lines were reversed, causing the order of firing to be reversed. When the problem was finally diagnosed, the plugs were all replaced with ones that supposedly _couldn't_ be reversed. An interesting side note: the low-tech means that the pilots developed to deal with the problem between the time they decided it _was_ a problem and the somewhat later time that it was "fixed" was to wire a pair of bayonets to the "rails" on either side of the ejection seat so that the points projected above the pilot's head. That way, if the seat blew before the canopy, the canopy would be shattered by the knives instead of by the pilot's head. Brent Chapman email@example.com or ucbvax!mica!chapman
The following short item appeared on p. 1 of the 25 November issue of _The_Wall_Street_Journal_; it's copied in its entirity, and as usual without permission: DON'T BLAME US. It's our naughty computer that keeps breaking the law. The law tells the IRS to wait 90 days after issuing a deficiency notice before trying to collect. If a taxpayer takes the matter to Tax Court, the IRS must hold off until the case is resolved. Yet, soon after Paul and Gina Husby proceeded to Tax Court, the IRS billed the San Francisco couple $47,000. A computer error, an IRS attorney assured them; don't worry. But more bills came, and the agency grabbed nearly $3,800 in Paul's credit union. Even after a federal court ordered a halt, the unlawful collection action went on. Finally, the IRS curbed itself, returned the credit union money, but argued the Husby's lawsuit for damages should be summarily dismissed. District Court Judge Weigel recently noted that the IRS "position boils down to the contention that the whole unfortunate incident was really nobody's fault, but the computers'." That won't let the agency off the hook, the judge said, ruling that the case should proceed to trial.
CALL FOR PAPERS FOR DEBUT ISSUE of THE JOURNAL OF COMPUTING AND SOCIETY Ablex Publishing Corporation will begin quarterly publication of a new academic journal on the social implications of computing technology in late 1988 or early 1989. The purpose of The Journal of Computing and Society will be to stimulate lively debate and speculation on a wide variety of topics concerning the computerization of society. The journal will be refereed. When it begins publication, individual subscriptions will be encouraged, and it should be available on stands in quality bookstores. Issues of the journal will be organized around themes. The first theme is "Has There Been A Computer Revolution?" Contributors are encouraged to submit papers relating to this theme (not necessarily answering the question yea or nay). Papers for this issue should be received by April 15, 1988. Future themes will include computers and war, computers and privacy, risks to the public, computers and power, the computer as metaphor, computers and gender, etc. Suggestions for themes are welcome, and the journal will publish a few articles outside the theme if the papers are of signficant quality, timeliness, and relevance to the purposes of the journal. Manuscripts should be 10-20 pages in length, conforming to the Chicago Manual of Style, and submitted in quadruplicate. A sheet of information for authors is available from the editor. The editor of this journal is Gary Chapman, executive director of Computer Professionals for Social Responsibility. Manuscript submissions and all correspondence should be directed to him at P.O. Box 717, Palo Alto, CA 94301. The telephone number is (415) 322-3778. The current editorial board of The Journal of Computing and Society consists of the following people: Jerry Berman Rob Kling Luca Simoncini Margaret Boden John Ladd Brian Smith David Burnham Abbe Mowshowitz Lucy Suchman Hubert Dreyfus Peter Neumann C.S. Tang Jean-Louis Gassee Susan H. Nycum Joseph Weizenbaum Calvin Gotlieb Kristen Nygaard Alan F. Westin Douglas Hofstadter Paul Saffo Langdon Winner Deborah Johnson Mike Sharples Terry Winograd Lenny Siegel
Please report problems with the web pages to the maintainer