The Risks Digest

The RISKS Digest

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Volume 1 Issue 22

Wednesday, 16 Oct 1985

Contents

oAdministratrivia
Friedrich von Henke
o Medical software incidents
Nancy Leveson
o European activities
Udo Voges
o Robots are different
Jerry Saltzer
o Automobile computer control systems
Bennett Smith
o Police computers
Dave Dyer
o Electronic Surveillance
Geoffrey S. Goodfellow / Bill Keefe
o Network Mailer Woes
Lynne Moore
o Databases, grades, etc.
Karl Kluge
Andy Mondore
Mark Sienkiew

Administratrivia

<>
From: Friedrich von Henke (vonHenke@SRI-CSL)

I am temporarily acting as guest moderator of the Risks Forum.
Peter Neumann had to go rather abruptly on an overseas trip, and
the transition happened a bit disorderly.  My apologies to all of
you who were eagerly awaiting their twice-weekly cost of RISKS readings
but had to go without.  I hope to have things under control now.

Apparently the hiatus has had the effect of slowing down the stream of
contributions to a merely trickle; please don't hesitate to get the flow
going again!

            Friedrich von Henke

----------------------------------------------------------------------

Date: 11 Oct 85 19:39:27 PDT (Fri)
From: Nancy Leveson <nancy@UCI-ICSD.ARPA>
To: RISKS@sri-csl.ARPA
Subject: medical software incidents

I was just on a panel concerned with Software Safety at an IEEE
conference on Computers in Medicine and heard about some more
incidents involving software faults.  

The first was cited in a recent RISKS forum message (about the
programmable implanted pacemaker which was inadvertently reprogrammed by
emitted magnetic fields from an anti-theft device in a retail store),
but the patient
was cited as having survived.  Unfortunately, his weakened heart was
unable to stand the increased pace, and he died.

Other recalls by the FDA involve:

1)  An infusion-pump (used for insulin) had a software problem which
caused the infusion of insulin or dextrose to be delivered at the
maximum rather than the lower intended rate.  This occurred when certain
valid data was entered according to user instructions.

2)  A programmable pacemaker "locked-up" when being reset by an
external programming device.  Luckily this occurred in a doctor's office,
and the doctor was able to revive the patient.

3)  A multiple-patient monitoring system was recalled because the software
got patients' names mixed up with the wrong data.

4)  An algorithm was incorrectly programmed in a diagnostic lab instrument
which caused certain patient data to be reported erroneously as all zeros.

The reference for these incidents (for those who want to quote them) is:
  H. Bassen, J. Silberberg, F. Houston, W. Knight, C. Christman, and
  M. Greberman.  "Computerized Medical Devices:  Usage Trends, Problems,
  and Safety Technology," in Proc. 7th Annual Conference of IEEE 
  Engineering in Medicine and Biology Society, Sept. 27-30, 1985, 
  Chicago, Illinois, pp. 180-185.

Nancy Leveson
University of Calif. Irvine


European activities

Udo Voges <voges@LOCUS.UCLA.EDU>
Fri, 11 Oct 85 11:38:57 PDT
I would like to bring some activities to your attention which are
going on in Europe, especially within and triggered by EWICS TC 7.

   The European Workshop on Industrial Computer Systems (EWICS), TC on
   Systems Reliability, Safety and Security (TC 7) is working since about
   10 years in this area, having some 100 members from industry, research
   and university. Previous work resulted in Position Papers on

       Development of safety related software
       Hardware of safe computer systems
       Guidelines for verification and validation of safety related software
       Guidelines for documentation of safety related computer systems
       Techniques for verification and validation of safety related software
       System requirements specification for safety related systems

   Current working areas include:

       System integrity
       Software quality assurance and metrics
       Design for system safety
       Reliability and safety assessment

   Besides conducting about four working meetings a year the TC is organizing
   the IFAC/IFIP Workshop on Achieving Safe Real-Time Computer Systems
   (Safecomp'79, '82, '83, '85, '86).

   The results of the work of TC 7 are introduced into the standardisation
   process (IEC, ISO, and national bodies) as well as used by companies
   and licensing authorities.

   Those interested in more information can either contact me or the
   current Chairman of TC 7: Mr. J.M.A. Rata, Electricite de France,
   1 Avenue du General de Gaulle,  F-92141 CLAMART   FRANCE.

   There exists an American counterpart to EWICS TC 7, but it was not
   possible to attract enough interested persons to keep it alive.
   The Japanese counterpart is also active, but due to the language
   barrier communication is minimal.

Udo


Robots are different

< Saltzer@MIT-MULTICS.ARPA>
Sat, 12 Oct 85 01:30 EDT
To:  risks@SRI-CSL.ARPA

When someone gets pinned to the wall by a robot, something different is going
on as compared to when someone gets gunned down by an FBI agent operating
under incorrect information retrieved from the NCIC.  Both cases may lead to
specific tragedies, yet the example of risks from robots seems to me to be
qualitatively different from many other computer-use risks.

The difference is that robots are used primarily in environments where
mechanically-oriented people are accustomed to balancing the risks of new
machinery against the benefits.  These people have, over the years, learned to
deal with risks from gear trains and drive belts, from swinging tailends on
steamshovels, from runaway elevators, from inadequately supported cranes.
They watch out, they learn from accidents, their insurers offer advice, they
make mistakes and take risks, and they learn.  To a first approximation, an
industrial robot presents a risk similar in kind to other new machinery, and
there is a moderately well-working system in place that is accustomed to
watching for the risks.  If anything, the average mechanic is suspicious of a
new piece of machinery in direct proportion to its complexity, newfangledness,
and gadgetry level, so is probably expecting the robot to screw up in
marvelous ways.  One might wish to argue with the particular balance that an
industry has struck between risks and benefits, but it is unusual to find one
in which mechanical risks are not understood at least moderately well.

The mechanic's suspicion of the new gadget is the essence of what seems to be
missing in many other applications of computers, and why it is so important to
raise awareness of the need to assess risks.  I'm not convinced we need to
harass our colleagues in the robot business with risk-consciousness-raising.
We should be instead talking to their installers to find out what we can
learn.

                                                  Jerry Saltzer


Automobile computer control systems susceptible to interference

Bennett Smith <ircam!bks@seismo.CSS.GOV >
Wed, 23 Oct 85 11:14:29 -0100
By chance I saw an article in an issue of the
"Journal of Environmental Engineers" (published in England, date of
issue about 10 months ago, I believe) about the sensitivity of
a microprocessor-controlled automobile control system to external
electromagnetic radiation.  As I recall, a CB transmitter near the car
could, at the right frequency, make the engine slow down or speed up.
Perhaps this article would interest some of your contributors.

Bennett Smith                   
IRCAM
31, Rue Saint Merri
75004 Paris, France     {seismo,philabs,decvax}!mcvax!ircam


The human element

<>
15 Oct 1985 23:42:01 PDT
From: Dave Dyer       <DDYER@USC-ISIB.ARPA>
To: risks@SRI-CSL.ARPA


 The human element really is where the action is, and it is
a completely two-edged sword;  Human actions which have the
power to "fix" something almost inherently also give the power
to "break" things equally severely.  Conversely, weighty
check and balance systems intended to prevent abuse end up
preserving the status quo, however good or bad that may be.

 The "police computer horror story" I'm most familiar
with is illustrative.  This is a well documented case
I've been reading about in ACLU publications.   

 It seems some poor soul had his wallet stolen, and some criminal
adopted his identity and later was involved in a robbery/murder.  
Through some circumstances peculiar to the case, the stolen
identity, but not the culprit, were known to the LAPD.  The
detectives working on the case put the stolen identity into
a national police computer.   Our hero was stopped for
a routine traffic citation, the computer coughed his 
name up, and he ended up on ice for a few days as a
murder suspect.  

  So far, this is pretty harmless and understandable.  Eventually
the guy's identity and and non-involvement were establishd and
he was turned loose.   Then it happened again.  And Again.
The guy began carrying a letter from the local chief of police,
saying he wasn't the guy the computer said was wanted, but
that didn't cut it when he traveled.

  The problem was that the LAPD detectives who put in the
original "want" refused to remove it.   Eventually the
guy (and the ACLU) got the courts to mandate expunging 
the computer.   I think the detectives involved and the 
LAPD are being sued.   Quite rightly.

  The point is, it is <<hard<> to design a system that
can do its intended job, permit discovery and correction
of errors, and resist unautherized or inappropriate use.
I can't imagine a system that can do all three.


Electronic Surveillance.

<>
24 Oct 1985 11:17-PDT
From: the tty of Geoffrey S. Goodfellow <Geoff@SRI-CSL.ARPA>
 
    [forwarded to RISKS by  Bill Keefe <keefe%milrat.DEC@decwrl.DEC.COM>
     from TELECOM Digest Volume 5, Issue 55, October 24, 1985]


Americans' Privacy Exposed by New Technology, Congress Told
 
By LEE BYRD - Associated Press Writer
 
    WASHINGTON (AP) - The explosion in communications technology has so
outpaced privacy laws that Americans have little or no protection
against a plethora of new ways for government or private adversaries
to pry into their lives, a congressional agency reported today.
    The non-partisan Office of Technology Assessment found that 35 out
of 142 domestic federal agencies use or plan to use various
electronic surveillance methods, including modern devices not
governed by a landmark 1968 law that circumscribed the use of
wiretaps and bugs - concealed microphones.
    The agency said 36 agencies, not counting those in foreign
intelligence, already use a total of 85 computerized record systems
for investigative or intelligence purposes, and maintain 288 million
files on 114 million people. The report raised the ''technically
feasible'' specter of these being linked into a single data base
network that could track untold numbers of citizens without due
cause.
    The report, requested by House and Senate committees, noted that
many new and uncontrolled methods of surveillance are made possible
by the very technologies of which more and more Americans are
availing themselves - electronic mail, computer conferencing,
cellular and cordless telephones, beepers and electronic pagers.
Intercepting such devices is easy, and ''the law has not kept pace,''
the agency said.
    But other devices, such as miniature television cameras and pen
registers - which monitor the numbers called on a given telephone
line - have enabled new ways to spy on people even if their own
communications habits are more old-fashioned, the agency noted.
    Rep. Robert W. Kastenmeier, D-Wis., chairman of the House Judiciary
subcommittee on courts and civil liberties, said the study ''shows
how the law in this area has broken down; it is up to Congress to fix
it. If we fail to act, the personal and business communications of
Americans will not have the privacy protection they deserve.''
    Sen. Charles McC. Mathias, R-Md., said the report ''documents how
new and more intrusive forms of snooping have followed in the wake of
the exciting advances in communications technology,'' and agreed
Congress must ''bring federal privacy laws up to date.'
    Rep. Don Edwards, D-Calif., chairman of the House Judiciary
subcommittee on civil and constitutional rights, said, ''While the
attorney general of the United States is claiming that the civil
liberties granted by the Constitution should be limited to the
'original intentions' of the framers, the technological possibilities
for government surveillance have exploded. The framers knew nothing
of closed-circuit television, wiretapping and computer data banks.''
    The report noted that the Fourth Amendment, which protects ''the
right of the people to be secure in their persons, houses, papers and
effects, against unreasonable searches and seizures,'' was written
''at a time when people conducted their affairs in a simple direct,
and personalized fashion.''
    Neither, said the report, has Title III of the Crime Control and
Safe Streets Act of 1968, which was designed to protect the privacy
of wire and oral communications, kept pace.
    ''At the time Congress passed this act,'' the report said,
''electronic surveillance was limited primarily to simple telephone
taps and concealed microphones. Since then, the basic communications
infrastructure in the United States has been in rapid technological
change.''
    The congressional agency said it could not estimate the extent of
electronic surveillance in the private sector, saying only ''it is
probable that many forms ... go undetected, and if detected, go
unreported.''
    But in its survey of the federal bureaucracy, OTA found 35 agencies,
mostly in the Justice, Treasury and Defense departments, used or
planned to use:
    -Closed circuit television, 29 agencies.
    -Night vision systems, 22.
    -Miniature transmitters, 21.
    -Electronic beepers and sensors, 15.
    -Telephone taps, recorders, and pen registers, 14.
    -Computer usage monitoring, 6.
    -Electronic mail monitoring, 6.
    -Cellular radio interception, 5.
    -Satellite interception, 4.
    As for the 85 computerized record systems that could be used for
surveillance purposes, none of the operators provided statistics
requested by the OTA on record completeness and accuracy.
    Under the 1968 law, wiretaps and bugs are prohibited without a court
order based on the affirmation of a high-ranking prosecutor that a
crime has occurred, that the target of the surveillance is involved,
and that other means of investigation would be ineffective.
    According to the Administrative Office of the U.S. Courts, federal
and state judges approved 801 out of 802 requests last year for
electronic surveillance, primarily wiretaps and hidden microphones,
at an average cost of $45,000.
    The agency said that while there is some promise in emerging
techniques for low-cost data encryption or other means to protect
communication systems from eavesdropping, ''there is no immediate
technological answer ... against electronic surveillance.''
    Foreign intelligence cases are governed by a separate law, so the
CIA, National Security Agency and Defense Intelligence Agency were
not included in the survey.
 

Network Mailer Woes...

"UV2::MOOREL" <moorel@uv2.decnet>
0 0 00:00:00 CDT [sic! ed.]
To: "risks" <risks@sri-csl>

    In a recent issue of one of the digests on the net, there was a problem
mentioned that seems to have a bearing on risks on computer systems, particu-
larly as use of computer networking increases in the future. At their request, 
the names have been changed to preserve anonymity.

            Apparently for the past month, the people who reside on the
    bitnet have been unable to receive [this digest].  There is a long
    story behind this [...]. This story is also a *lot*
    of guesswork as to what happened.
            At the beginning of September, [our system] changed its host 
    name to conform to the new domain name standards.  The site we
    were using to get to bitnet, did not recognize the new name and
    began rejecting all mail from [our system].  We did not become
    aware of this because we were not receiving any rejections or errors
    back from the gateway.  We were however, receiving mail *from* the 
    people on Bitnet who were asking what happened to their [...] digest.
            We attempted to contact the people at [the gateway] but of 
    course the mail failed and they never did anything to correct the 
    problem which they, of course, were not aware of because nobody was
    complaining.  (Sounds like a Catch-22 situation if I ever heard
    one).
            In any case, the problem has now been resolved.
    Unfortunately, these people have missed close to 50 digests.  There
    is no way I can tie up the mailer at [either the host or the gateway] 
    in order to remail the messages.  I also understand that there is no 
    way to use FTP from the bitnet. 

It appears that several incidents conspired together to cause the loss of this 
information, and although the loss was not critical, it will take much time and
effort for the people involved to catch up. If this had been a more critical 
information transfer, it might have been corrected faster; however, there is a 
good chance that it would have gone undetected anyway. It seems to be one more 
reason for information about any potential changes to be passed on to any sites
that may be affected and to be thoroughly checked on both ends to prevent this 
kind of thing from reoccurring.

    Lynne C. Moore (Moorel@Eglin-Vax.Arpa)


Grade changing.

<Karl.Kluge@G.CS.CMU.EDU>
Some doubt has been expressed in this forum recently about people
changing grades and living happily ever after. I can't talk about
college systems, but in high school all the grades, attendence, etc.
for my high school and several other high schools were kept on a
mainframe in a centralized location. There is a system in Michigan
called MOIS for vocational data, and on the back of my MOIScript on
computer science was the transcript of a terminal session between the
attendance people and the computer. The login message gave the phone
number of the mainframe. The password was overprinted, but that is
useless -- you can learn to read through almost any overprinting.
Access to the grading, course scheduling, and attendance programs was by
providing a social security number which was echoed and not overprinted.
I thus found myself able to do really miraculous things. Being sixth in
my high-school class, I had no real motivation to use my knowledge
maliciously, and informed the administration. Some safeguards were added
(old social security numbers retired, certain social security numbers
only giving access to certain programs, restricting access to certain
programs to certain accounts), but I'm sure those could have been
circumvented with minimal effort -- I was a fairly good systems hack on
the operating system, and there were people who could hack rings around
me. The operating system was a simple three-tier ring system, and a lot
of the features put in for the sake of usability made it very insecure.

I send this to give first-hand evidence to those who have been posting
doubts that such things happen.


Databases, grades, etc.

<Andy_Mondore%RPI-MTS.Mailnet@MIT-MULTICS.ARPA>
Wed, 16 Oct 85 17:11:36 EDT
One of the systems programmers here at RPI made a point about
administrators and students sharing the same computer:  You
really aren't that much more secure when you have administrators
and students on separate computers if there is a network or dial-up
connection to the administrative computer than you are when
administrators and students are on the same computer.  If you have
network or dial-up connections to an administrative computer, it
isn't too difficult for a student with an autodial modem to write
a program on a PC that simply tries all possible phone numbers
for certain telephone exchanges and record the numbers that
produce a carrier tone.  Then the student could have another
program that tries passwords unless the system disconnects the
line after a certain number of unsuccessful tries.  The major
advantage of separating administrators and students is that
it might be more difficult for a student to access an administrative
file from a student account assuming the administrative file has the
proper protection set.


Database, Grades, etc...

< Sienkiew@louie.udel.EDU>
Mon, 21 Oct 85 0:44:24 EDT
You can create an extremely effective audit trail if you think about it for a
while.  That just makes the problem more "challenging".

Suppose you do your auditing one day and find that there were unauthorized
grade changes made for every student in the CS department and 1/2 of them
requested (incorrect) printed transcripts.  It seems unlikely that everybody
independently broke in on the same day.  So who do you penalize?  How many
transcripts have been mailed out already?

Suppose no grades were changed but there is a trojan horse waiting to raise
the grade only under certain circumstances?

My point is that the data doesn't have to stay changed forever.  And you can't
check the auditing records for every transaction, if you expect to gain by
using the computer.  You need to do as much as you can, of course, but beware
of the false sense of security...

            Mark.

Please report problems with the web pages to the maintainer

Top