Please try the URL privacy information feature enabled by clicking the flashlight icon above. This will reveal two icons after each link the body of the digest. The shield takes you to a breakdown of Terms of Service for the site - however only a small number of sites are covered at the moment. The flashlight take you to an analysis of the various trackers etc. that the linked site delivers. Please let the website maintainer know if you find this useful or not. As a RISKS reader, you will probably not be surprised by what is revealed…
Readers of RISKs might be interested in a recent Oxford University Press paperback book "Benefits and Risks of Knowledge-Based Systems" (ISBN 0-19-584743-9). This is a 76-page Report of a Working Party of the (UK-based) Council for Science and Society, chaired by Professor Margaret Boden (Prof of Philosophy and Psychology at the School of Cognitive and Computing Sciences, University of Sussex). The main chapter headings are: 1 How Knowledge-Based Systems Work 2 Applications of KBSs 3 The Benefits and Dangers of KBSs 4 Influencing the Future Uses of KBSs 5 Conclusions and Recommendations Summarizing the final conclusions and recommendations: 1 The general level of public awareness about the applications and social implications of KBS is low, and education about KBS should be included in school courses. 2. There should be more interdisciplinary undergraduate and postgraduate study programs in cognitive sciences and KBSs. 3. A government initiative applying KBSs to health education and health care would be popular and in the long term cost-effective. 4. A KBS should complement human workers rather than replace them. 5. It is undesirable to substitute a computer for a human function, such as the giving of psychiatric help, that should involve respect, understanding, empathy or love between humans. 6. Great attention should be given to the immense hazards presented by possible malfunctioning of military command and control systems and autonomous decision-making programs. 7. The Data Protection Act should entitle people to know not only what data are held about them, but also the rules by which they are processed. 8. Statutary regulation of KBS standards may be needed in future. 9. KBS vendors should adopt a suitable Code of Practice. 10. Meanwhile they should be legally obliged to insure themselves against claims for damages by users. 11 Various industry standards are needed, especially in the interface between the program and the user. 12 Professional associations such as the Society for the Study of Artificial Intelligence and the Simulation of Behaviour, and the Expert Systems Group of the BCS should adopt a code of conduct for their members. 13 Setting up a formal body for KBS practitioners with restricted membership is *not* recommended. Brian Randell, Computing Laboratory, University of Newcastle upon Tyne, UK EMAIL = Brian.Randell@newcastle.ac.uk PHONE = +44 91 222 7923 FAX = +44 91 222 8232
I am reposting this, after being told that my original postings were only seen
on BITNET. Sorry BITNET readers for a possible quadruple posting :-(
[It has also been posted on SWL-L.]
---------------------------Original message----------------------------
A posting appeared recently on the SWL-L list regarding a computerized
information system that informs motorists about traffic conditions:
>Here in Chicago, the Illinois Department of Transportation operates a
>group of low-power AM stations that provide traffic information.
>
>The stations broadcast at the top and bottom of the dial (540 and 1610
>I think). The broadcast alternates between events affecting traffic
>and actual traffic reports. It is all automated.
>
>There is a tape that describes the daily schedule for construction
>work, lane closures, ramp closures, parades, and other events
>affecting traffic.
>
>This alternates with computerized traffic reports: "As of ... 5:55 pm
>... there is severe congestion on the following currently monitored
>sections of the highway system ... Kennedy expressway ... inbound ...
>between Montrose avenue ... and Irving Park road ... between Kedzie
>avenue ... and the Ohio street junction ..." and so on. The reports
>are updated every 10 minutes or so by sensors buried in the highways.
>
>[text omitted]
>
>One problem with this system is that when traffic comes to a complete
>stop somewhere, the computer doesn't detect cars moving over the
>sensors and decides that there is no traffic. This is fairly rare.
>[text omitted]
>
>It's a system that I like a lot.
I find it amusing that this user likes the system, despite its apparently
well-known tendency to be completely wrong in worst-case conditions. I
wonder if people from out of town are told about this quirk, or left to
discover it on their own.
>EMAIL: draughn@iitmax.iit.edu+--------------+ Academic Computing Center
>BITNET: SYSMARK@IITVAX | Mark Draughn | 10 W. 31st St.
>VOICE: +1 312 567 5962 +--------------+ Illinois Institute of Technology
>ALSO: ...{nucsrl|att}!iitmax!draughn Chicago, Illinois 60616
Today I called my bank (Bank of the West) to transfer some money from one
account to another. Before initiating the transfer, I asked for the balance in
the source account. The amount the teller said was in my account seemed
somewhat high, but in the right ballpark. Since I hadn't balanced my checkbook
recently and since it covered the amount I was about to transfer out, I
initiated the transfer, thanked the teller, and hung up.
Immediately after getting off the phone, I balanced my checkbook, and realized
that the amount she had read to me was about $2K too high (though the correct
amount still covered the transfer). I called back immediately, this time
getting a different bank worker. I told her that the amount I had been given
earlier was incorrect. She said that the bank had just installed "new
software" the previous day, and all account balances were temporarily wrong
until "things sort themselves out", which would be a day or so. She said that,
unfortunately, not all the bank's employees were aware of the problem.
I wonder what "until things sort themselves out" means. "Until all
transactions recorded in the old system are entered into the new one", perhaps?
If so, why didn't they do this before putting the new system on-line to
workers? Maybe "until bugs in the software are fixed?" If so, their testing
of the new software was seriously inadequate.
JJ
The RSRE Viper microprocessor and Avra Cohn's report on its formal verification, have been discussed earlier in RISKs. Readers may therefore be interested in the following article, by Simon Hill, which appeared on p.3 of the (UK) Computer Weekly for July 5, 1990. It is reprinted here in its entirety, without permission. Brian Randell, Computing Laboratory, University of Newcastle upon Tyne, UK :: USER THREATENS COURT ACTION OVER MoD CHIP :: :: The first commercial user of the Viper safety-critical chip developed :: by the Ministry of Defence is threatening legal action for alleged :: misrepresentation. :: :: Teknis International Railroad Systems of Adelaide, Australia, is :: seeking assurances that the Viper technology can meet the claims that :: the MoD and its commercial partners make for it. :: :: Teknis, which is developing a signal and railway crossing control :: system using Viper for the Australian National Railway Commission, is :: also threatening action against the MoD's commercial licensee, Charter :: Technologies. :: :: Worcester-based Charter was licensed in january 1988 to exploit :: commercially the fruits of the Viper work carried out at the Royal :: Signals and Radar Establishment at Malvern. :: :: Ron Davison, Teknis' business development manager, says, *We are :: looking for every comfort we can get from the development and :: suppliers of Viper:. :: :: Davison says the A$12m Australian railways project "is a world first" :: in the safety-critical market, making the first time that Viper has :: found a user outside the military and defence communities. :: :: Teknis' concern has been inspired by a series of reports in UK and US :: academic circles about RSRE and Charter's claims that Viper is :: formally verified for use in safety-critical applications where lives :: may be put at risk if the technology fails. :: :: Davison says he is "surprised at the sudden rash of reports about :: Viper coming out of the woodwork" 18 months after Teknis began work :: with the chip. :: :: But the report that is most critical of Viper, written by Avra Cohn of :: Cambridge University's computer laboratory, is two years old. It was :: published in May 1988 and delivered to RSRE, but Charter technologies :: claims it was not shown Cohn's findings until mid-1989. :: :: RSRE and Charter claim that Viper is formally specified, with a chip :: design which conforms to this specification. Cohn says in the report :: that this is misleading. :: :: "Such assertations, taken as assurances of the impossibility of design :: failure in safety-critical applications, could have catastrophic :: results," Cohn says in the report. :: :: The MoD says "It is a matter of interpretation of the words used to :: describe the dependabiliity of Viper. Nothing can be described as :: absolutely fail safe." :: :: This year a report by US consultants Computational Logic for US space :: agency Nasa says "Viper has not been formally verified" and lists four :: deficiences in RSRE's specification. In a draft copy of the same :: report dated June 1989, obtained by Computer Weekly, the former chief :: RSRE scientist on the Viper projects, John Cullyer, has indicated his :: agreement with Nasa's conclusions. Cullyer is now Professor of :: Electronics at Warwick University. :: :: The MoD cannot say whether the Nasa and Cohn reports have been looked :: at by RSRE staff, but a spokesman says, "Work is continuing to :: reinforce verification techniques and if a relevant report has been :: produced then it will be studied by scientists at RSRE." :: :: Marconi Electronic Devices of Lincoln, sub-contracted by the MoD to :: manufacture Viper hardware circuitry, is reining back on its :: commitment to the project while it waits for replies from the MoD. :: :: Tony Smith, Marconi Electronic Devices' integrated circuits contract :: manager, says the company "wanted a discussion with MoD and RSRE about :: what could be guaranteed for Viper. That meeting was due to take :: place this year, but the MoD cancelled it. We have still not had that :: meeting". :: :: Marconi has asked the MoD to respond to the Cohn and Nasa reports, but :: has not yet received a reply and has not been shown either of the :: reports, Smith says. The company is making prototype Viper circuits, :: but has no commercial orders. :: :: The Ministry of Defence would not comment on "confidential or :: commercial correspondence between it and third parties". :: :: The MoD says, "No Viper chip is known to have failed, but work is :: continuing to reinforce and improve verification techniques: on Viper, :: and that *although there are not known faults in the Viper design, an :: unremitting search for weakness must continue". :: ::
The following appears in an article advising against the use of cellular telephones in aircraft (it's illegal in most countries), published in Flight Safety Bulletin (the quarterly journal of the General Aviation Safety Committee in the UK): "As a point of interest, the manufacturers of some mobile telephones have issued a warning that they should not be used in cars fitted with an electronic anti-lock braking system (ABS) because the cellphone could cause the system to malfunction." Presumably the risk is from EMI. Does any RISKS reader have details on this? Is it just a precautionary warning to limit legal liability, or is there evidence? If this is a real risk, what about coaches, trains, other nearby vehicles ...? Martyn Thomas, Praxis plc, 20 Manvers Street, Bath BA1 1PX UK. Tel: +44-225-444700. Email: mct@praxis.co.uk
The following is taken from an article by Kevin Eason in the (London) Times of
July 9th:
A safety alert has been issued to thousands of mobile telephone
engineers after an incident in which a Jaguar car lost all power
because of faulty telephone installation. The driver escaped
safely when all electrical systems, including lights, brakes and
engine, were shut down by a crossed wire.
Jaguar drivers were warned yesterday to use only car phones
fitted and checked by company dealers. The Federation of
Communications Services has alerted its 350-member companies
which handle mobile communications equipment that faulty
installation could damage vital equipment, especially anti-lock
braking systems (ABS), and urges them to check with manufacturer
specifications.
[...]
The Jaguar driver involved was stranded on a motorway when all
power to the car failed. Checks on the vehicle showed that a
telephone engineer had crossed vital wires which caused a
complete systems shutdown.
Most new cars now have complex computer engine management systems
and increasing numbers have electronic controls for suspension,
brakes, gearbox and safety mechanisms. The new Mercedes SL
convertible sports car, for example, has an automatic pop-up roll
bar operated by computer.
[...]
Tim Duncan, AI Applications Institute, University of Edinburgh,
80 South Bridge, Edinburgh EH1 1HN, Scotland, United Kingdom.
JANET: T.Duncan@uk.ac.edinburgh
ARPA: T.Duncan%uk.ac.ed@nsfnet-relay.ac.uk
BITNET: T.Duncan%uk.ac.ed@ukacrl.bitnet
UUCP: ... mcvax!ukc!ed.ac.uk!T.Duncan
I unfortunately had a Hertz rental car on Orcas Island, Washington,
when a Washington State Ferry rammed the Island's only car ferry dock and
knocked it out of service for a minimum of several days. The incident raised
several interesting points:
1) Hertz had no category for this case; they repeatedly told
us "If the car were broken, then you could fly off the island and get
another car, and retrieving your car would be our problem. But since
you say it isn't BROKEN, every day your car sits on the island is a day
you are renting it... and of course if we have to pick it up there you get
a retrieval fee, a fee for renting one-way, etc etc".
2) It would be interesting to get more informed information about
how the ferries work. From the June 29, 1990 Seattle Times:
"All indications point to a failure in the vessel's computer-control
system. Inspectors will attempt to recreate the situation which caused the
computer failure, but 'often these computer glitches have a tendency not to
reappear'. ... The six Issaquah-class ferries ... were built in the late '70s
and early '80s and have been plagued by propulsion-control problems. The
system's top engineer in 1986 ... said then that the propulsion systems
were so unpredictable that if he had his way he would replace them. Last
year the state awarded a $550,000 contract ... to do design work for the
ferry system, including modifications on the Issaquah-class propulsion-
control system. ... Unlike the original propulsion system which controls
and reverses the variable-pitch propellers on the diesel-engine ferries
through a computer, the new system is simpler and more reliable ... ."
From what I could gather from talking to the Seattle Times reporters
covering the incident, the ferries have a "fly by wire" propulsion system that
among other things does automatic docking. Occasionally the propulsion-control
computer hiccups, locking the controls for a few seconds. Normally this is no
bid deal, but in this case the computer system had the bad timing to hiccup
just as the ferry was supposed to start braking for the final stage of docking
at Orcas. As a result the ferry didn't brake at all and ploughed straight into
the terminal, causing $250,000 in damage. (Also costing island merchants much
of their July 4 weekend tourist revenue, and severely inconveniencing many
unlucky tourists. We eventually managed to get our Hertz car off the island
by private barge.)
Driving to work this morning there was an interesting story on the radio. It seems the Domino's Pizza joint closest to the Pentagon can accurately predict when a major operation is about to take place. Evidently the planning meetings go on long into the night, and the best place to get food is Domino's. They interviewed someone from Domino's and he said that prior to the Panama invasion deliveries to the Pentagon jumped 25%. I'm glad I'm not in security.....
This is from the July 19 issue of Computer Weekly:
The 9 billion pound Trident nuclear warhead programme is being held up
because of serious problems with a computer system, developed by Logica,
that keeps highly volatile nuclear materials apart during weapon assembly.
The system, which ensures the safe movement and records the exact
whereabouts of plutonium and other materials around two buildings at the
Ministry of Defence's Atomic Weapons Establishment (AWE) at Aldermaston,
is only able to handle half the volumes set by the MoD. Despite this, the
system went live on July 13.
[...]
The system is a high integrity movement control package, using Stratus
fault-tolerant hardware, used in A45 and A90, the two buildings at AWE
involved in Trident development work.
One source says that the MoD has set a target for the system to handle
200 units of nuclear material within the two buildings at any one time,
but that the present system can only handle 100 units.
The source says that the MoD and AWE have been revising the target
downwards, but that Logica has been unable to meet the new figure.
[...]
A Logica spokeswoman says it is not company policy to comment on
projects which have not been announced.
Yes, this is the same Logica that ran into trouble during a project for
San Fransisco's transit system.
Mark Smith smith@canon.co.uk
Canon Research Centre Europe ..ukc!uos-ee!canon!smith
According to a story by Jeffrey A. Perlman in the 'Los Angeles Times' 18 July 1990, the General Accounting Office has issued a new report entitled "Air Traffic control — Inadequate Planning Increases Risk of Computer Failures in Los Angeles." The GAO says that in planning for a 1995 consolidation of Southern California radar tracking facilities, FAA officials have refused to consider computer solutions that could solve their computer problems because the agency does not want to rewrite or develop software to run on new, state-of-the-art hardware and expend the "additional time they believe would be required to undertake such an effort." A 1989 GAO report singled out the Los Angeles - Orange County air space as having the worst comuter-related aircraft tracking problems. The new report criticizes the FAA for ignoring the situation, saying air traffic controllers may find their radar screens flickering, showing insufficient data or blanking out at critical moments. FAA officials "are still analyzing the GAO report and had no formal reply."
A short item in 'Business Week', July 30:
BMW PUTS A BACKSEAT DRIVER ON A CHIP
The latest cars offer a plethora of computerized wonders — like chips that
control the engine or warn you is a door is ajar. But would anyone want a
computer that took over control of a car if it judged the vehicle was being
driven too fast or improperly for road conditions? BMW engineers are
betting the answer is yes.
They've already installed an early version of their Heading Control system
in cars at the auto company's research center in Munich. A camera above
the rearview mirror tracks the center stripe and the line along the right
side of the road. If a driver gets too close to either marker, a small
electric motor integrated into the steering system is activated to put
things right. Later versions will gauge road conditions and differentiate
between broken and solid lines, so the computer can tell such things as
whether it's okay to pass. Drivers being corrected might feel a tug on the
wheel. But they can easily override the computer by continuing with
wahtever they are doing. BMW engineers say the system is at least five
years from market. And they predict that once customers get used to the
idea, they'll love it.
>From the Reuters news wire:
BRITISH AIR DEFENSE COMPUTER SUFFERING A "NERVOUS BREAKDOWN"
LONDON, Reuter - A nearly half-billion dollar computer
designed to mastermind Britain's future air defenses won't be
fit for action for another 10 years because of bouts of nervous
confusion, Defense Ministry officials said Wednesday.
The $448.8 million system known as ICCS — Improved UK Air
Defense Ground Environment Command and Control System — was due
to enter service in 1987. But officials told Parliament's
defense committee the computer's problems with logic would delay
operation by up to 10 years.
"We get wrong answers sometimes and the bugs have to be
tracked down and sorted out," Donald Spiers, who heads aircraft
control at the ministry, told the committee.
"Secondly, from time to time it crashes. What this means is
the equivalent of a nervous breakdown. It becomes confused with
the information and goes wrong."
The production consortium, made up of Hughes Aircraft,
Marconi and Siemens-Plessey, have upgraded the main computers
driving the system to give more power, Spiers said.
ICCS was designed as the sophisticated equivalent of the war
tables used during World War II when model planes were pushed
around a board.
It will coordinate radar, fighters, surface-to-air missiles
and air command centers through a single network.
The system is being developed on a fixed price contract and
Spiers said all extra costs had to be met by the consortium.
He said the companies had not been paid for the past two
years because of the problems encountered.
--Walt Thode Internet: thode@nprdc.navy.mil
UUCP: {everywhere_else}!ucsd!nprdc!thode
>From the Guardian, Mon. 23rd July, front page: Headline: BR signalmen 'worked blind' Subhead: Computer software problems admitted at key commuter train centre By-line: Patrick Donovan, Transport Editor British Rail has admitted that computer software problems have been uncovered at a signal centre which controls London's busiest commuter lines. They left operators "working blind" after train movements were wiped off control screens on at least one occasion over the last five weeks. A BR spokesman said newly installed software, responsible for flashing up the position of trains on the indicator screens of signal operators at Wimbledon, has been found to contain two technical faults. The Wimbledon centre controls 90 mph services south of Waterloo and includes the Clapham Junction area, where 35 people died in a train accident in December 1988. Faulty wiring on a signalling modernisation programme was found to have caused the crash. BR said one of the faults uncovered on the indicator screen software has not yet been fully rectified. An internal investigation began after an operator found that the system was providing "the wrong information". Realising that he had lost track of train movements, the operator immediately turned all signals to red. A spokesman said that at no time was any train at risk. "What happened caused concern to the signalman." But he stressed that the mechanical signal equipment and all other equipment worked normally, bringing all trains to an immediate standstill after the problem was discovered. "The problem was caused by computer software fault in the signal box. [sic - PM] It gave the wrong indication to the signal man. All the trains froze where they were. The lights told him that something was different to what was happening [outside]." BR conceded that the faulty equipment served a vital function, "this little piece of software tells the signalman what is happening outside". The software faults were found inside the panel in the train indicator box in a system responsible for operating the lights. Alastair Wilson, contracts and production director of E. B. Signal, the manufacturers, said: "The system is under test. I do emphasise that things are going through a testing stage. It is not unusual to have minor software bugs." [!!! - PM] A spokesman for the National Union of Railwaymen said that any operational shutdown of train indicator screens would "at best create a major disruption and at worst could create alarming safety hazards. If everything goes to red it puts enormous pressures on an individual signalman." [Remainder of article omitted. It goes on about unrelated signalling problems in the Clapham Junction area, including a loss of communication between Waterloo and Wimbledon signal centres, concern about the growing use of driver-only services on Network SouthEast, contradictions in the rule-book about whether or not such services may operate with a defective radio, and the need for the driver to draft competent members of the public to assist him in ensuring safety after an emergency stop.] A few comments: The report is rather confused, but it seems that the system receives information about the position of every train in the region, and displays this to the signalmen in the form of lights moving over a large indicator board. The bugs were in the hardware/software module which drives the display, so it was a case of "correct information in, garbage out". I wonder if the faulty module was rated as "safety-critical" in the hazard analysis. It would be nice to know *how* the signalman knew that he was being given wrong information, and what would have happened if he had not been so alert, and continued to operate the network with the wrong information. It is amusing to hear the manufacturer refer to a bug which brings an entire railway regional network to a standstill for some time as "minor". Perhaps he meant that only a few lines of code were in error! Somebody should have a word with him about how to classify faults and failures! If, as the manufacturer states, the system is under test, why was it being run to control live traffic without any back-up system? Surely BR's testing strategy can't be "Let the train take the strain"? (To quote one of their own advertising slogans!) Peter Mellor, Centre for Software Reliability, City University, Northampton Square, London EC1V 0HB Tel.: +44 (0)71-253-4399 Ext. 4162/3/1
CALL FOR PAPERS
Symposium on Testing, Analysis and Verification
Victoria, British Columbia, Canada October 8-10, 1991
Sponsored by ACM Sigsoft
The purpose of this meeting is to bring together researchers and
practitioners who are working in the areas of software analysis,
testing, and formal verification. Papers and panel session proposals
are invited on current and emerging techniques, strategies, processes
and tools for determining the presence or absence of errors in software
and for inferring other characteristics of software quality.
Papers should be a maximum of 5000 words in length. They should present
a clear picture of the original contributions made by the paper, while
also carefully relating the work presented to the work of others. The
highest quality papers from the symposium may be considered for
publication in a special issue or section of a research journal.
Authors should send six copies of their paper or panel proposal to the
Program Committee Chair, Nancy Leveson, at the address below.
Papers must be received by March 1, 1991.
Authors will be notified of acceptance by May 15, 1991.
Camera-ready papers are due not later than July 1, 1991.
Program Committee
Victor Basili Mark Moriconi Stephen Schach
Lori Clarke Mitsura Ohba Gene Spafford
John Gannon Tom Ostrand K.C. Tai
Susan Gerhart Dewayne Perry Elaine Weyuker
Carlo Ghezzi Richard Platek Jack Wileden
Richard Hamlet Debra Richardson Bill Young
John Knight John Rushby Michal Young
For other information concerning the symposium contact:
General Chair Program Chair
Prof. William Howden Prof. Nancy Leveson
Computer Science Dept. Computer Science Dept
University of California University of California
La Jolla, California 92093 Irvine, California 92717
USA USA
(619) 534-2723 (714) 856-5517
Please report problems with the web pages to the maintainer