[Regarding ``sanity checks''] Let us remember that there are sane ``sanity checks'' as well as the other kind. About 8 years ago while a grad student at an Ohio university that probably ought to remain unnamed, I learned of the following follies: The campus had long been doing class registration and scheduling via computer, but the registrar insisted on a ``sanity check'' in the form of hard copy. Once each term, a dozen guys in overalls would spend the day hauling a room full of paper boxes over from the CS center, representing a paper copy of each document that had anything to do with the registration process. [I first took exception to this because their whole argument in favor of "computerizing" was based on reduced costs, but I guess that should be hashed out in NET.TREE-EATERS.] No one in that registrar's office was at all interested in wading through all that paper. Not even a little bit. One fine day, the Burroughs people came through with a little upgrade to the processor used by campus administration. And some "unused status bits" happened to float the other way. This was right before the preregistration documents were run, and dutifully about 12,000 students preregistration requests were scheduled and mailed back to them. All of them were signed up "PASS/FAIL". This was meticulously recorded on all those trees stored in the back room, but no one wanted to look. I suppose a moral would be ``if you include sanity checks, make sure a sane person would be interested in looking at them.'' [Regarding break-ins at Stanford] A lot of the discussion seems to revolve about ``hey, Brian, you got what you asked for'' (no matter how kindly it is phrased). Without making further editorial either way, I'd like to make sure that Brian is commended for sharing the experience. Sure would be a shame if ``coming clean'' about a bad situation will be viewed as itself constituting a risk... [I am delighted to see this comment. Thanks, Brian! PGN]
The intuitive sense made by Dave Benson's argument in RISKS 3.50, that >We need to understand that the more faults found at any stage to >engineering software the less confidence one has in the final product. >The more faults found, the higher the likelihood that faults remain. seems to invite a search for confirming data because there are also counter- intuitive possibilities. For example there is the notion that the earlier in the life cycle errors are detected, the cheaper to remedy them. There is a premium on finding faults early. And the further notion that with tools for writing requirements in some kind of formal language that can be checked for syntactic and semantic completeness and consistency, it's possible to detect at least some errors at requirements stage that may not have been caught till later. So SE projects using these and similar methods for other stages in the life cycle would tend to show more errors earlier. Would the products from these projects be therefore less reliable than others made with, say, more traditional, less careful, design and programming practice ? Dave makes the further argument in RISKS 3.57: >Certain models of software failure place increased "reliability" on >software which has been exercised for long periods without fault. [...] The models of software reliability exist to order our thinking about reliability and to help predict behavior of software systems based on observation of failure up to the current time. The models that show failures clustered early in time and then tapering off later do indeed model an intuition but maybe not the one that more faults mean yet more faults. Hence the need for data. I suspect that the reality as shown by data, if it exists, would be more complex than intuition allows. More errors discovered so far may just mean better software engineering methods. As far as other engineering fields, the failure vs time curve in manufactured products is often taken to be tub-shaped, not exponentially decaying. So more failures are expected at the beginning and near the end of the useful life of a "hard" engineered product. Of course, "an unending sequence of irremediable faults" should be the kiss of death for any product, whether from hard engineering or soft. But the trick is in knowing that the sequence is unending. The B-17, I seem to remember reading, had a rather rocky development road in the 1930s, yet was not abandoned. Was it just that the aeronautical engineers at Boeing then had in mind some limit on the number of faults and that this limit was not exceeded? It might be easy to say in hindsight. On the other hand, sometimes foresight, in terms of spotting a poor design at the outset, makes a difference, as in the only Chernobyl-type power reactor outside the Soviet block. It was bought by Finland (perhaps this is what "Finlandization" means ?). However the Finns also bought a containment building from Westinghouse. Ken Dymond
[The two participants requested this clarification be included for the record... PGN] You misinterpreted my message in a small way; I was writing about a university attended by a friend, NOT Rice university. To my knowledge, Rice has been very good about protecting its students' privacy. My student number is NOT my social security number, though the university has that number for good reasons. I do not want anyone to think that I was talking about Rice. David
[THE FOLLOWING DISCOURSE INVOLVING CLIFF AND HERB IS LIKELY TO CONTINUE FOR A WHILE ON ARMS-D. PLEASE RESPOND TO HERB LIN, NOT TO RISKS ON THIS ONE. HERB HAS VOLUNTEERED TO SUBMODERATE, AND THEN SUBMIT THE RESULTS TO RISKS. PGN] From: Clifford Johnson <GA.CJJ at Forsythe.Stanford.Edu> An "autonomous weapon" [should be] defined to be any weapons system which is de facto preprogrammed to take decisions which, under the law of nations, require the exercise of political or military discretion. It's not a bad first attempt, and I think it is necessary to get a handle on this. With the realization that you have done us a service in proposing your definition, let me comment on it. I don't understand what it means for a weapon to "take a decision". Clearly you don't intend to include a depth charge set to explode at a certain depth, and yet a depth charge could "decide" to explode at 100 feet given certain input. What I think you object to is the "preprogrammed" nature of a weapon, in which a chip is giving arming, targeting and firing orders rather than a human being. What should be the role of the human being in war? I would think the most basic function is to decide what targets should be attacked. Thus, one modification to your definition is An "autonomous weapon" [should be] defined to be any weapons system which is preprogrammed to SELECT targets. This would include things like roving robot anti-tank jeeps, and exclude the operation of LOW for the strategic forces. But this definition would also exclude "fire-and-forget" weapons, and I'm not sure I want to do that. I want human DESIGNATION of a target but I don't want the human being to remain exposed to enemy fire after he has done so. Thus, a second modification is An "autonomous weapon" [should be] defined to be any weapons system which is preprogrammed to SELECT targets in the absence of direct and immediate human intervention. But then I note what a recent contributor said -- MINES are autonomous weapons, and I don't want to get rid of mines either, since I regard mines as a defensive weapon par excellence. Do I add mobility to the definition? I don't know.
There's great difficulty in defining "autonomous weapons" so as to separate some element that seems intuitively "horrible" about robot-decided death. But a workable definition is necessary if, as CPSR tentatively proposes, such weapons are to be declared illegal under international law, as have chemical and nuclear weapons. (Yes, the U.N. has declared even the possession of nukes illegal, but it's not a binding provision.) The problem is, of course, that many presently "acceptable" weapons already indiscrminately-discriminate targets, e.g. target-seeking munitions and even passive mines. Weapons kill, and civilians get killed too, that's war. Is there an element exclusive to computerized weapons that is meaningful? I don't have an answer, but feel the answer must be yes. I proffer two difficult lines of reasoning, derived from the philosophy of automatic decisionmaking rather than extant weapon systems. First, weapon control systems that may automatically target-select among options based upon a utility function (point score) that weighs killing people against destroying hardware would seem especially unconscionable. Second, but this presumes a meaningful definition of "escalation," any weapons system that has the capability to automatically escalate a conflict - and is conditionally programmed to do so - would also seem unconscionable. Into the first bracket would conceivably fall battle management software and war games, into the second would fall war tools that in operation (de facto) would take decisions which according to military regulations would otherwise have required the exercise of discretion by a military commander or politician. The latter category would embrace booby-trap devices activated in peacetime, such as mines and LOWCs; and here there is the precedent of law which prohibits booby traps which threaten innocents in peacetime. Perhaps the following "definition" could stand alone as *the* definition of autonomous weapons to be banned: An "autonomous weapon" is defined to be any weapons system which is de facto preprogrammed to take decisions which, under the law of nations, require the exercise of political or military discretion. This might seem to beg the question, but it could be effective - military manuals and international custom is often explicit on each commanders' degree of authority/responsibility, and resolving whether a particular weapon was autonomous would then be a CASE-BY-CASE DETERMINATION. Note that this could, and would, vary with the sphere of application of the weapons system. This is reasonable, just as there are circumstances in which blockades or mining is "legal" and "illegal." Of course, a case-in-point would be needed to launch the definition. Obviously, I would propose that LOWCs were illegal. How about battle management software which decides to engage seemingly threatening entities regardless of flag, in air or by sea? Any other suggestions? Does anyone have any better ideas for a definition?
In thinking about this question, I believe that ARMS-D and RISKS could perform a real service to the defense community. There is obviously a concern among some ARMS-D and RISKS readers that autonomous weapons are dangerous generically, and maybe they should be subject to some legal restrictions. Others are perhaps less opposed to the idea. It is my own feeling that autonomous weapons could pose the same danger to humanity that chemical or biological warfare pose, though they may be militarily effective under certain circumstances. I propose that the readership take up the questions posed by Cliff's recent contribution: What is a good definition of an autonomous weapon? What restrictions should be placed on autonomous weapons, and why? How might such limits be verified? Under what circumstances would autonomous weapons be militarily useful? Should we be pursuing such weapons at all? How close to production and deployment of such weapons are we? Maybe a paper could be generated for publication?
Please report problems with the web pages to the maintainer