The long running Alcor/email case against the County and City of Riverside, CA was settled out of court in April of this year. The announcement was delayed until all parties had signed off, and the check had cleared the bank :-). The Alcor Life Extension Foundation (a non-profit cryonics organization --firstname.lastname@example.org) ran a BBS for members and prospective members from early 1987 through January 12, 1988. On that day, the BBS computer was removed under a warrant to take the computer (but no mention of any contained email) in connection with the investigation into the death of 83-year-old Dora Kent. (Mrs. Kent was placed into cryonic suspension by Alcor in December of 1987. During and following the investigation, Alcor staff members were publicly accused by county officials of murder, theft, and building code violations. No charges were ever filed and the investigation was officially closed three years later.) In December of 1988 Keith Henson filed a civil suit to force an investigation of the apparent violations of the Electronic Communication Privacy Act by the FBI, but the case was dismissed by the now convicted Judge Aguilar. In early 1990, just before the statute of limitations ran out, Henson and 14 others (of the roughly 50 people who had email on the system) filed a civil action against a number of officials and the County and City of Riverside, CA under Section 2707 of the Electronic Communication Privacy Act which forbids inspecting or denying access to email without a warrant. Some time after the case was filed, the Electronic Frontier Foundation came into existence in response to law enforcement abuses involving a wide spectrum of the online community. EFF considered this case an important one, and helped the plaintiffs in the case by locating pro bono legal help. While the case was being transferred, the County and City offered a settlement which was close to the maximum damages which could have been obtained at trial. Although no precedent was set because the case did not go to trial, considerable legal research has been done, and one judgment issued in response to the Defendants' Motion to Dismiss. The legal filings and the responses they generated from the law firm representing the County/City and officials are available by email from email@example.com or (with delay) from firstname.lastname@example.org. (They are also posted on Portal.) The Plaintiffs were represented by Christopher Ashworth of Garfield, Tepper, Ashworth and Epstein in Los Angeles (408-277-1981). The only significant item in the settlement agreement was the $30k payment to the plaintiffs.
The 12 June 1991 issue of JAMA (the Journal of the American Medical Association) contains an important article and editorial. Article: Blumberg Mark S (1991). Biased Estimates of Expected Acute Myocardial Infarction Mortality Using MedisGroups Admission Severity Groups. JAMA 265 (22): 2965-2970. Editorial: Iezzoni Lisa I (1991). 'Black Box' Medical Information Systems: A Techology Needing Assessment. JAMA 265 (22): 3006-3007. Blumberg evaluated the actual mortality in a group of patients and compared those values with the predictions of a standardized computer model which is mandated for use in stratifying Medicare-aged patients with acute myocardial infarction (AMI, i.e. heart attack) in Pennsylvania and (with modifications) in other states. The stratification of patients is important for evaluating the quality of care. If your hospital sees mostly older patients with multiple chronic diseases, your mortality associated with AMI will be greater than that found in a hospital which treats generally healthier patients. The use of a complex model to adjust for the differences in patient populations between hospitals should, in an abstract sense, leave only those differences which relate to quality of care factors specific to an individual hospital and allow people (mainly health care administration and hospital accreditation bureaus) to rate the hospitals for quality. In these days of increasing health care regulation, attempts to define quality by creating models which normalize mortality (or some other measure) as a means of comparing hospitals (or doctors, surgical procedures, methods of treatment, etc.) are being introduced more and more often. The system Blumberg tested depends on the evaluation of multiple key clinical factors (KCFs) which are coded by technicians from the medical record and run through a proprietary computer program resulting in assignment of the patient to a risk class. Note that the system is far to complex to permit any individual to assess the correctness of its predictions. In addition, the system is proprietary and so the algorithm and coding scheme is not generally available. [Even if it were, it is doubtful that anyone outside the corporation that developed it would be able to comprehend it.] Blumberg found that the classification method is statistically biased and concluded that the bias arose from a number of sources (e.g. missing KCFs, KCFs which can vary a great deal in a short time, failure to distinguish missing from normal data, and incorrect weighting). This bias has significant implications for hospitals and physicians and (by extension) for patients. Since the output of the system is used as a measure of quality, some hospitals look better than others for reasons unrelated to the quality of care. There is an incentive to try to improve the 'numbers', i.e. to restructure care to make the quality measure appear more attractive [although doing this is made difficult by the fact that it is hard to determine what controls the evaluation system's output]. In an accompanying editorial, Iezzoni notes that the system is an example of a broad class of health care information product "about which very little is known: the redoubtable 'black box'." She goes on to point out that these systems "typically derive from fields about which users may have little knowledge, such as health services research. Many systems base their determinations on complicated statistical formulas that are dependent on the computational wizardry of today's powerful personal computers. Hence, even if complete technical information is available concerning the system's internal algorithm, it often remains an enigma to those whom it affects... Another factors fostering the black box mystique is the proprietary nature of some of these information products, and the consequent reluctance to reveal 'inner workings' and perceived 'trade secrets'." She then notes that there are no applicable standards which require that these systems be safe and effective (in a manner analogous to the requirements for new drugs) and observes that "[f]ew independent evaluations have been performed, and little is known about their validity for use in new policy initiatives, such as statewide hospital quality assessment. While most developers are at least rhetorically committed to improving the health care provision system, this commitment is not enough, given the tremendous responsibility vested in these systems. Opening the black box is only the first step. We need to know that the information generated is valid and used in a safe and effective way." The article and editorial face squarely what has been dealt with in only a peripheral manner before: whatever the _theoretical_ potential for producing effective decision support systems using computer models to compare different groups or organizations, the _practical_ effect of such systems is likely to be quite problematic. (1) Because the systems represent such a large computation, there is no practical way in which users of the systems can understand their output. These systems are _oracular_, that is, their output stands apart from their input in incomprehensible ways (just like the pronouncements from the Oracle at Delphi: there was no conceivable way to know if it they were correct). Some black box systems are comprehensible. We use a number of measuring devices whose internal workings are not easily understood (e.g. the pulse oximeter used to measure oxygen saturation in the operating room) but these are mostly black boxes whose operation is meaningfully connected the observable (and observed) world and whose performance can be understood by individuals. When the black box reaches a certain size or when its performance is determined by lots of data or data over a long period of time, it's output necessarily becomes oracular. The new TCASS aviation collision avoidance system is likely to be a case in point. (2) Because of the great effort to required to generate and integrate these systems (many man years, including massive data collection) it is practically impossible to test them in a meaningful way. Blumberg's effort is one example, but he tested just one medical diagnosis out of many supported by the system; it's hard to think of someone going on to test all the others. (3) The systems depend on information which is easy to acquire (e.g. review of the patient's chart) but may be only remotely related to the issues at hand (e.g. quality of care). The systems have a kind of self qualifying nature: the choice of data and pragmatics of acquisition and processing limit the validity of the conclusions in ways which most of us will agree are unrealistic and misrepresent important features and characteristics of the real underlying process, yet there are no easy ways to improve the data. (4) The systems are always potentially open to successive refinement but their size and complexity makes such refinement unlikely. The makers of such systems will always admit that they are not perfect but that the imperfections could be reduced through improvements or modifications. In reality, however, the system is more likely to remain largely fixed or to track changes in the world only slowly. (5) Paradoxically, these systems may obscure the search for quality by providing an objective, computationally efficient, quantitative metric which is related to quality in only a complicated and remote way. In developing "safety" or "quality assurance" systems, there is a tendency to redefine the goal in terms of the system's output. What are the implications of this trend in health care (and other domain) information systems for professionals developing decision support systems? When Iezzoni refers to "powerful personal computers" she surely means the computers and those system designers, integrators, programmers, coders, and administrators that make these complex systems possible. What are the responsibilities of designers of such decision support systems? After all, they are well situated to assess the impact of decision support tools on the performance of the large system. Do they perhaps have a responsibility to refrain from creating a large complex information processing system which is likely to have the qualities of the one investigated by Blumberg? Copyright (c) 1991 by Richard I. Cook
While reading through the source code to the Berkeley Time Server (which runs in the background of a group of Unix workstations and keeps their system clocks adjusted to "network average time"), I discovered an interesting code sequence in the networkdelta function. That function takes a set of time delay measurements and computes the network average time change. I.e., it is the core of the time server algorithm: /* this piece of code is critical: DO NOT TOUCH IT */ ... i++ if (i = j) j++; ... Those of you familiar with C programming will recognize the classic error (I make it frequently) of writing "i = j" (assignment) rather than "i == j" (equality test) in an if statement. Both are legal in this context: "i = j" meaning "assign the value of j to i and then test for inequality to zero." Some reflections: -- Burying an erroneous statement in a paragraph that says "don't touch" makes matters worse. I only found the bug when I went back to 30 year old "Math 295" tools of pencil and paper and walked through the algorithm one statement at a time to see how it worked. -- The error will only manifest itself if one or more systems is wildly out of agreement with the other systems being served by the time daemon. I.e., it is a classic "normal error" in that it is triggered by some other error and makes matters worse. -- The error results in an incorrect calculation of the network average time, which will be corrected (if anyone notices it) when the entire network is re-synchronized to a standard clock (several dial-up time standard clocks are readily accessible from a dial-up modem). -- If Berkeley (the copyright holder) didn't distribute source code, I wouldn't have found the error. Instead, I'd have written my own procedure which almost certainly have been a poorer algorithm. -- Code reviews — having your software carefully reviewed by a competent outside consultant — are useful. (What is the computer engineering equivalent of a pathologist?) -- Beware of language constructions, such as C's "if (i = j)" that are error prone. Having once tried to add a warning for this to a C compiler, I can attest to it being extremely difficult: you want to warn on "if (i = j)" but not on "if ((i = j) != 0)" Ultimately, I decided the cure (heuristics in a the compiler) might be worse than the disease. -- "Beware of language constructions" is a warning to the programmer, and one that belongs in a "Manual of Programming Style." It is an engineering statement, not one of "Computer Science." I.e., it is at a different level of discourse than "beware of bubble-sorts." -- My university sent me to a remedial writing course because I couldn't spell or distinguish between "who" and "whom," not to mention "that" and "which" — should there be remedial programming courses, taught by writing teachers, that concentrated only on style? It is possible to write quality bug-free software in error-prone languages (just as it is possible to write poor software in languages that would prevent "if (i = j)" errors). However, I am beginning to suspect that this requires the obsessive attention to detail of a contract lawyer, combined with the grace of expression of an essayist. Martin Minow
>>>>> In <a href="/Risks/12/02">RISKS 12.02</a>, 3 Jul 91 03:29:41 GMT, Fernando Pereira said: F> [A subsequent revised version of the AP story summarized above reports F> on speculation that the cause of the phone disruptions may be sabotage F> originating in the Middle East. The alleged reason for this is the claim F> that in most cases the network failures followed the appearance of F> animated hieroglyphics on operators's terminals.] More likely when the big crash came an ESC ( } or ESC ( 0 or the like came over the line (along with other accidental garbage characters), common ways of turning on many terminals' "line drawing character set" etc. Good thing the terminals didn't also have a alternative Russian character set, or else there seems a slight RISK that we might be fighting World War III now.
On Chaos Computer Club's last Congress 1990, a Dutch group and few other phreaks reported on some techniques to "travel inexpensively on international networks" (see my report in January 1991). Against their usually detailed description of the content of the respective session, CCCs electronic Congress newspaper describes the reports and discussion only in general terms; no details regarding frequencies and computer programs (which meanwhile replaced the "blue boxes" more flexibly) were given. According to a report in the ("usually well-informed") German weekly magazine Der SPIEGEL, the Dutch group HAC-TIC now published a detailed report on how to "use" special methods, dial-tunes (with frequencies and sequences of operation) and telephone numbers (in Germany: 0130) in diverse areas of the world to establish toll-free phone connections via specific programs. As the magazine reports, HAC-TIC aims with its detailed description to counterfeit some people who sell (e.g. on AMIGA) such tune-dialing programs for up to 1,000 DM (about 520$ currently). Comment: In discussing with CCC people about their surprisingly careful publication behaviour (enough details to warn before developments, but not sufficient to directly aid in attacking), I found some response to the international discussion on CCC related attacks; against CCCs behaviour in earlier years (e.g. selling the NASA attack protocols for 100 DM), this restrictive policy seemed quite honestly. Now that HAC-TIC has published details of the seminar discussion, another discussion might well come in CCC whether such a restrictive approach was well-advised in CCCs spirit. Klaus Brunnstein, University of Hamburg, FRG
As an enthusiatic reader of the Risks Forum, I strongly appreciate the information carried and the discussion usually maintained on a high level of competence. Moreover, the diversity and actuality of the topics highly demonstrates that if Risks Forum had not been invented by PGN it must be now. So, RF is a success! Unfortunately, in discussing critical developments as well as preparing IFIP 1992 World Conference (Madrid, September 7-11, 1992) substream "Diminishing Vulnerability of Information Society", I find it ever more difficult to contribute some essential information which I earlier would have immediately sent. One example: since some time, we experiment how to detect and prevent worm accidents; to this end, we had to analyze minimum requirements of worm mechanisms. Our experiments in NETBIOS and DECNET environment showed clearly that it will be *extremely difficult* to prevent or even detect worms! In this situation, I need start a discussion in a responsive and responsible community how to proceed from here; such a discussion would need more knowledge to assess the threat. At this point, I learn from several discussion in universities and conferences that Risks Forum is now well-spread and even attracted the interest of scenes such as Chaos Computer Club or diverse BBS read by youngsters. Not that I know yet of a single case where actual discussions in Risks Forum has produced a new threat; but when observing other electronic newsmedia, I find it highly desirable that a discussion of the limit of electronic discussion of risks be undertaken. My examples are from the anti/virus scene: recently a discussion started (in Virus-L) on how to propagate viruses when invoking commands such as DIR; this technology is well established in MACviruses but virtually unknown in the MsDos arena; what is the impact of such discussion between experts on the virus authors? Another example: When observing recent developments in new threats, our analysis shows too clearly that their authors know the actual discussion rather well; example: the TEQUILA virus exhibits several aspects of older, less distributed viruses which have been described in detail in Virus-L. Therefore, any constructive discussion in publicly accessible electronic newsmedia must bare in mind the risk to transfer serious information to the wrong parties. Evidently, insecurity and insafety is an essential message which Risks Forum communicates to the public. This mission is an essential contribution of the ACM committee behind RF (whose chairman Peter is), and as IFIP TC-9 "Computer and Society" chairman, I heavily favour any such discussion. On the other side, responsible behaviour also questions whether reports on computer-induced vulnerability generates, by too much details, new vulnerabilities thus endangering the developments of methods to counterfeit vulnerability. I personally think that a responsible message must be more complex than just arguing: "information technology produces vulnerability; when eliminating IT, you immediately reduce vulnerability": such as simple suggestion is in-historic. My suggestion for a Code of Discourse Ethics: 1) Concerned experts should agree not to enhance the vulnerability by their discussion. 2) apart from questions of experiences and basic paradigms, aspects of prevention and countermeasures should even more be discussed. 3) For critical technical details, successful electronic media are not well suited, even they are counterproductive. To avoid any misunderstanding: the guarantee of "free flow of information" is one of the essential values in modern societies, specially IT-based ones. But the "trust" which I assume my communication partners follow may not simply be established via electronic media; trust (defined differently from TCSEC contexts!) is a personal relation to minimize the risk of misinterpretation and the deduction of unwished consequences. By its very nature, trust is hardly to be established via email! The medium therefore limits the responsible use of it. (Example: I personally just received Bill Gates memo on Microsoft's performance and future problems; highly interesting, no doubt: but I assume that Bill Gates will not be glad that I had it. I am highly sure that the community in which I received this information is trustable, and they and I will not uncover any details; but just the fact that I as a non-Microsoft employee got it demonstrates the problem!). If anybody of the highly respected active participants in this discussion feels this discussion inadequate, I apologize for stimulating it. Klaus Brunnstein, University of Hamburg, Germany
>This kind of thinking is, unfortunately, all too common, even in the scientific >community. If I disagree with it, it must be wrong. If it supports what I >believe, it must be right. I'm not sure of the relevance of this note to computing risks, but I am offended by the use of a borderline dishonest debating trick in which one mis-attributes an obviously absurd opinion to an opponent. Which members of the "environmental movement" have have exhibited unlimited confidence in weather forecasting? One can certainly be alarmed at the prospect of global warming while maintaining a great deal of skepticism about the current state of the art in weather prediction. Again, I'm not sure why this material found its way into the RISKS digest, but I don't want to leave it unchallenged.
[ To the moderator: I don't really get the joke here, and I'm not sure the reference to the risk of uncritical acceptance of simulation results justifies opening this can of worms in this forum, but I don't think it would be appropriate to let Reisman's trivialization of real problems go unanswered. --crk ] In RISKS 12.01, Rob Slade quoted a poster at the University of Michigan who quoted George Reisman's "The Toxicity of Environmentalism": >The environmental movement maintains that science and technology cannot be >relied upon to build a safe atomic power plant, to produce a pesticide that is >safe, or even bake a loaf of bread that is safe, if that loaf of bread contains >chemical preservatives. > >When it comes to global warming, however, it turns out >that there is one area in which the environmental movement displays the most >breathtaking confidence in the reliability of science and technology, >[ in the area of very-long-term weather prediction ]. Aside from my reservations about Reisman's rhetorical trick of taking examples from the views of extremists to trivialize a broad-based movement with many thoughtful adherents, this analysis ignores well-founded concerns. Goods and services aren't produced by disinterested scientists in a laboratory. They're produced by real-world corporations that have to balance risk against cost. In many well-documented instances, the financial stakes have been so high that real risks have been covered up and safety improvements have been foregone. The best example of this is the nuclear power industry, in which it's difficult to make any operational regulation more stringent because to do so would be to acknowledge that previous regulations had been inadequate and that currently-operating plants are less safe than is technically possible. In cases like this and like global warming where the perceived hazard is extreme, the public is unwilling to accept grandfather clauses and assurances that "this subject needs further study; we're not sure what to do yet". I'll argue that the public concern over global warming is less a result of trust of simulations than of distrust of the technologies that are being blamed for the process. As for the punchline about predicting the weather, I doubt that the current level of concern over global warming would have arisen except that the last decade has shown us (at least in North America) the warmest weather in the past century and a half. The argument that this warm trend is not yet statistically significant as evidence of greenhouse-gas-caused global warming is less persuasive than is personal experience that the weather is warmer than it used to be. -- Chuck Karish email@example.com Mindcraft, Inc. (415) 323-9000
In RISKS DIGEST 12.02 <firstname.lastname@example.org> (The Polymath) wrote an anecdote on how an old risks posting had been resurrected for posterity in an upcoming book. It is precisely these activities that make RISKS one of the most valued forms of electronic publication. A RISKS posting can find its way from the electronic media into the paper form (via SIGSOFT notices and CACM) quite rapidly. These items can then be cited by other workers in the field. It is this phenomenon that makes RISKS almost accepted as a form of refereed publication. It is the efforts of our moderator that makes this so. [Included in all modesty... Keep the good stuff coming. PGN]
Please report problems with the web pages to the maintainer