The RISKS Digest
Volume 30 Issue 89

Tuesday, 30th October 2018

Forum on Risks to the Public in Computers and Related Systems

ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Please try the URL privacy information feature enabled by clicking the flashlight icon above. This will reveal two icons after each link the body of the digest. The shield takes you to a breakdown of Terms of Service for the site - however only a small number of sites are covered at the moment. The flashlight take you to an analysis of the various trackers etc. that the linked site delivers. Please let the website maintainer know if you find this useful or not. As a RISKS reader, you will probably not be surprised by what is revealed…

Contents

MTR East Rail disruption caused by failure of both primary and backup
Hong Kong Free Press
Train stops in exactly the wrong place
Mark Brader
Texas straight-ticket voters report ballot concerns
Arthur Flatau
MikeA
Australian risks of voting systems
Sheldon
Re: U.S. Begins First Cyberoperation Against Russia Aimed at Protecting Elections
Monty Solomon
Tech support—Hubble telescope
Rob Slade
Login glitch behind Tokyo Stock Exchange snafu
Nikkei Asian Review
State surveillance company leaked its own data, its customers' data, and its customers' victims' data
BoingBoing
"New Windows 10 1809 bug: Zip data-loss flaw is months old but Microsoft missed it"
Liam Tung via Gene Wirchenko
Car Interfaces
Gabe Goldberg
Driverless cars: Who should die in a crash?
bbc.com
Every minute for three months, GM secretly gathered data on 90,000 drivers' radio-listening habits and locations
BoingBoing
Surgery students 'losing dexterity to stitch patients'
bbc.com
In Cyberwar, There are No Rules
Foreign Policy
Lawmakers Seek Review of Pentagon Contract Thought to Favor Amazon
WiReD
The customer is always right ... re: Apple iPhones
Rob Slade
Fun with source code
Medium
A Dark Consensus About Screens and Kids Begins to Emerge in Silicon Valley
The New York Times
When Trump Phones Friends, the Chinese and the Russians Listen and Learn
NYTimes
Apple appears to have blocked GrayKey iPhone hacking tool
Lucas Mearian
Re: Toward Human-Understandable, Explainable AI
DJC
Re: Explainable AI Simulation for AVs
Richard Stein
Info on RISKS (comp.risks)

MTR East Rail disruption caused by failure of both primary and backup (Hong Kong Free Press)

Richard Stein <rmstein@ieee.org>
Mon, 29 Oct 2018 22:06:46 +0800
https://www.hongkongfp.com/2018/01/11/mtr-east-rail-disruption-caused-failure-primary-backup-servers/


Train stops in exactly the wrong place (Modern Railways)

Mark Brader <msb@vex.net>
Mon, 29 Oct 2018 14:56:29 -0400
According to a short item on page 87 of the October issue of "Modern
Railways", on August 21 a suspected shoplifter was chased into a train
tunnel at Amsterdam's Schiphol Airport, requiring the train service to be
temporarily shut down.  But when they went to restart it, the entire
computerized train management system crashed and would not come back up.  As
a result, all trains throughout the greater Amsterdam area were halted from
some time in the evening rush hour until after midnight when the bug was
finally identified and fixed.

"It transpired", the article says, "that one train had been stopped
at exactly the point where the software determines which platform a
train should use" and hence "the software continuously detected a train
arriving at the spot and proceeded to try and allocate the non-existent
arrival (the train was already there!) 32,000 times before the system
crashed."


Texas straight-ticket voters report ballot concerns

Arthur Flatau <flataua@acm.org>
Sat, 27 Oct 2018 08:07:15 -0500
Austin American Statesman

The idea that using hitting a button or other control while a screen is
rendering is a user error is astounding.  If the machine incorrectly
interprets user input it is a bug plain and simple.

Amid scattered complaints by straight-ticket early voters of both parties
that their ballots did not, at first, correctly record their choice of
either Democrat Beto O'Rourke or Republican Ted Cruz for U.S. Senate, state
and local election officials are cautioning voters to take their time in
voting and check the review screen for accuracy before casting ballots.

The elections officials say the problems resulted from user error in voting
on the Hart eSlate machines widely used in Texas—including in Travis,
Hays and Comal counties—and are not the result of a machine glitch or
malfunction.

“The Hart eSlate machines are not malfunctioning,'' said Sam Taylor,
communications director for the Texas secretary of state's office.  “The
problems being reported are a result of user error—usually voters hitting
a button or using the selection wheel before the screen is finished
rendering.''

Taylor said the office is aware of a handful of complaints and that the
voters were able to correct their ballots before casting their votes.

https://www.statesman.com/news/20181026/texas-straight-ticket-voters-report-ballot-concerns

  [On the other hand, this explanation might be somewhat evasive.  For
  example, see Kim Zetter' article on this subject: Voters in Texas aren't
  to blame for vote-switching in Cruz/O'Rourke race; a software issue known
  as a race condition or concurrency bug is, says Dan Wallach, who notes
  machine vendor failed to fix this and many other problems found with the
  Hart machines at least ten years ago.
  https://twitter.com/KimZetter/status/1057332585313910785

  Note: Dan Wallach, Rebecca Mercuri, and I testified before the Houston
  City Council in July 2001 on why the these machines (still in use today)
  were likely to be vulnerable.  PGN]


Texas straight-ticket voters report ballot concerns (RISKS-30.89)

mikea <mikea@mikea.ath.cx>
Thu, 25 Oct 2018 20:59:15 -0500
People have been talking about voting machines registering a vote other than
the one the voter intended. It happened to a friend in Collin County, Texas.
She voted Straight Democratic Party on an electronic voting machine, and had
her votes change to all Republican candidates for the same positions. It was
good that she noticed this before she actually hit the button to register
her votes. She noticed that the process was repeatable: straignt Democratic
party changed to straight Republican party a second time, called an election
judge over, and demonstrated it a third time.

The election judge reluctantly took that voting machine out of service.

I find myself wondering if the same thing happened to others who *didn't*
notice before they completed the vote using that machine.

My more paranoid self, noting that these machines have no paper ballots as a
permanent record, wonders if the machine was somehow rigged to change straignt
Democratic to straight Republican—the more so because Collin County is
pure, saturated RGB=(255,0,0) Republican. It also wonders how many more
machines did the same change.

My _extremely_ paranoid self wonders if there are documents circulating
among a small subset of election officials, with titles like "How to rig
FooCorp voting machines to help your side".

An acquaintance who works for the election board in a Georgia county tells me
that the reports that votes for the Democratic candidate for Governor were, at
the ultimate moment being changed *in the voting machine* to votes for the
Republican candidate—again, on all-electronic machines that dont use paper
ballots and have no audit trail.

Paper ballots make true recounts possible. Who controls these voting machines
controls the election.


Australian risks of voting systems (RISKS-30.88)

Sheldon <sheldon10101@gmail.com>
Tue, 23 Oct 2018 22:44:19 -0400
The Australian experience with counting votes will not work for the US.
I've been a DRO, someone who has run a poll, at Canadian Federal, Provincial
and Municipal Elections.

Counting by hand the less than 200 ballots for a Federal or Provincial
election was no problem. There is a paper ballot and one office to count. I
told the scrutineers (partisans who watched the count) that they had a few
seconds to look at a ballot and object. Then, I'd decide.  If they didn't
like the decision, that ballot went an envelope for disputed ballots along
with spoiled ballots. In case the vote was very, very close, they first
looked at those questionable ballots.I was one of the first to get my ballot
box back to the riding office.

Counting by hand a municipal election where there were two different ballots
and 5 offices on a ballot was a nightmare. After doing one, I never did
another one. Now there are still two different ballots but, the ballots are
counted by OCR.

The Election lists are maintained by a non-partisan body. There are ID
requirements but, with the liberals in power, very little is required.  In
the past, the position of election officials on the day of the election was
partisan. Now, they are happy to take anyone.

Of course, with the mad Doug Ford in power in Ontario, no one knows where
his madness will lead.  Ontario elected an idiot knowing he was an idiot.
We just didn't know how much of an idiot he would be.


Re: U.S. Begins First Cyberoperation Against Russia Aimed at Protecting Elections (Solomon, RISKS-30.84)

Richard Stein <rmstein@ieee.org>
Wed, 24 Oct 2018 18:06:58 +0800
https://techcrunch.com/2018/10/23/first-cyber-operation-gentle-approach-russian-trolls/

A line in a CV stating: "Recipient of US Cyber Command email advising to
cease and desist election interference, and immediately end trolling in
OCT2018" must be an honor among the Russian cyberwarrior cognoscenti.

RISK: Does it justify a salary raise request?


Tech support—Hubble telescope

Rob Slade <rmslade@shaw.ca>
Thu, 25 Oct 2018 12:10:07 -0700
Two weeks ago, the Hubble telescope experienced a gyroscope failure.

Hubble has been very important, and has contributed enormously to our
understanding of the universe.  This is a hugely expensive device, which has
had problems in the past.  It's up in space where you can't exactly get
someone to go and hit it with a hammer in hopes it'll start working again.

NASA has tried a number of sophisticated procedures to get Hubble
functioning again.  They haven't worked.

Now NASA has turned it off, and back on again.
https://gizmodo.com/hubble-telescope-s-broken-gyroscope-seemingly-fixed-aft-1829934018 or
https://is.gd/JgwOMu

Hubble is working again ...

When I'm dying in hosptial I want them to unplug all the tubes and plug them
back in and see if that works ...


Login glitch behind Tokyo Stock Exchange snafu (Nikkei Asian Review)

Gabe Goldberg <gabe@gabegold.com>
Tue, 30 Oct 2018 14:58:54 -0400
https://asia.nikkei.com/Business/Markets/Login-glitch-behind-Tokyo-Stock-Exchange-snafu


State surveillance company leaked its own data, its customers' data, and its customers' victims' data (BoingBoing)

Lauren Weinstein <lauren@vortex.com>
Wed, 24 Oct 2018 11:44:41 -0700
via NNSquad
https://boingboing.net/2018/10/24/20-gb-of-internal-data.html


"New Windows 10 1809 bug: Zip data-loss flaw is months old but Microsoft missed it"

Gene Wirchenko <genew@telus.net>
Tue, 23 Oct 2018 18:31:07 -0700
Liam Tung, ZDNet, 23 Oct 2018

https://www.zdnet.com/article/new-windows-10-1809-bug-zip-data-loss-flaw-is-months-old-but-microsoft-missed-it/

A Feedback Hub user reported the latest Windows 10 October 2018 Update bug
three months ago. Microsoft has fixed the issue in preview builds of the
19H1 version of Windows 10, so it should be fixed in 1809 soon.

opening text:

Windows 10 version 1809 update is still on ice due to the data-deletion bug
embarrassingly missed by Microsoft during preview testing.

But the few users who did get the Windows 10 October 2018 Update have now
discovered its built-in zip tool is doing weird things when copying files.

As one 1809 user reported on Reddit, this version of Windows 10 is missing
the 'Do you want to replace these files' dialog when copying from a zip
archive to a folder with an identically named file in it.

The problem only seems to affect the built-in zip tool in Windows File
Explorer rather than third-party zip tools.

The dialog is an important flag when transferring a lot of files, since it's
an opportunity for the user to choose whether to replace the identical file,
skip replacing the file, or compare the information stored in both files
before taking any action.

Without the dialog, it could be easy to unintentionally overwrite
non-identical files.


Gabe Goldberg <gabe@gabegold.com>
Thu, 25 Oct 2018 15:29:00 -0400
Switches and dials have been the norm for controlling things in cars, from
the side mirrors to audio volume. But norms evolve. As automakers prepare
for a world of shared self-driving cars, they're experimenting with an array
of human-machine interface technologies, or HMIs, including interior-facing
cameras, gesture and voice controls, and touch-sensitive surfaces ” all
augmented by ever-smarter computing platforms.

Voice controls are en route to be the second most-prevalent interface by
2022, when it's forecast to be in 80 percent of car HMIs, up from 48 percent
in 2016, according to the consulting firm Frost & Sullivan. Data published
last year in the firm's Global Connected Car Market Outlook show
touchscreens on top, with 90 percent market share by 2022, up from 29
percent two years ago. Multifunctional controllers (50 percent from 16
percent), handwriting recognition (30 percent from nine percent), digital
instrument clusters (25 percent from seven percent) and head-up displays or
HUDs (20 percent from five percent) follow. Only gesture controls will
remain relatively rare in four years, with just five percent HMI penetration
worldwide, but still up tremendously from 0.02 percent in 2016, Frost &
Sullivan predicts.

They're helping the driver "get more accustomed to newer technologies, so
that the user acceptance is there before he or she is going to give over
control to the car in autonomous mode," says Niranjan Manohar, research
manager for connected car and automotive IoT (Internet of Things) at Frost &
Sullivan in Detroit.

https://www.cta.tech/News/i3/Articles/2018/September-October/Human-Machine-Interfaces-Evolve-in-Cars.aspx


Driverless cars: Who should die in a crash? (bbc.com)

Richard Stein <rmstein@ieee.org>
Sun, 28 Oct 2018 12:51:47 +0800
https://www.bbc.com/news/technology-45991093

"To get closer to an answer - if that were ever possible - researchers from
the MIT Media Lab have analysed more than 40 million responses to an
experiment they launched in 2014.

"Their Moral Machine has revealed how attitudes differ across the world."

With a software update, an AV "born" in China can be tuned for trolley
problem "death" preferences anywhere, just like language locales for
international-friendly applications. All the AV needs to know, per the
"Moral Machine," are passenger/occupant ages and species.

RISK: Does the AV have the "right" to act on its own volition if there are
no human occupants or the passenger "species" are marginalized (insects or
bacteria)?


Every minute for three months, GM secretly gathered data on 90,000 drivers' radio-listening habits and locations (BoingBoing)

Lauren Weinstein <lauren@vortex.com>
Tue, 23 Oct 2018 11:11:03 -0700
via NNSquad
https://boingboing.net/2018/10/23/dont-touch-that-dial.html

  On September 12th, GM's director of global digital transformation Saejin
  Park gave a presentation to the Association of National Advertisers in
  which he described how the company had secretly gathered data on the
  radio-listening habits of 90,000 GM owners in LA and Chicago for three
  months in 2017, tracking what stations they listened to and for how long,
  and where they were at the time; this data was covertly exfiltrated from
  the cars by means of their built-in wifi.  The company says it never sold
  this data, but the presentation to the advertising execs was clearly
  designed to elicit bids for it.

Unless they had explicit fully-informed consent from drivers, this
should be—and may have been—illegal!


Surgery students 'losing dexterity to stitch patients' (bbc.com)

Richard Stein <rmstein@ieee.org>
Tue, 30 Oct 2018 10:53:50 +0800
https://www.bbc.com/news/education-46019429

"A professor of surgery says students have spent so much time in front of
screens and so little time using their hands that they have lost the
dexterity for stitching or sewing up patients."

Western medical training today emphasizes computer simulation over the
"human touch" to learn the art. Simulated triage procedure rehearsals,
especially from mass shooting incidents or industrial accidents, can help
prepare medical team readiness.

Would a surgical patient feel reassured to know that their physician learned
colectomy or appendectomy exclusively by computer simulation rather than
acquired via hands-on experience?

Should surgeons be required to publicly disclose performance statistics: #
of hours simulation practice for specific surgery, # of hands-on vs. robot
surgery assists, # of computer-assist fatalities and incidents, etc.?

Intuitive Surgical can cite this article to promote their da Vinci Surgical
System.


In Cyberwar, There are No Rules (Foreign Policy)

Richard Stein <rmstein@ieee.org>
Fri, 26 Oct 2018 10:55:32 +0800
https://foreignpolicy.com/2018/09/12/in-cyberwar-there-are-no-rules-cybersecurity-war-defense/

"If a country or terrorist group decided to take out a sitting U.S. senator
undergoing robotically assisted surgery and then covered its tracks, the
perpetrator's identity would be hard to pinpoint, and there would be no
clear U.S. legal precedent for classifying the hacking of hospital equipment
as an assassination or an act of war. Nor do there appear to be clear
protocols for retaliation."

A verifiable cyberweapons treaty urgently required to establish rules of
conduct and preempt escalation.


Lawmakers Seek Review of Pentagon Contract Thought to Favor Amazon (WiReD)

Gabe Goldberg <gabe@gabegold.com>
Sun, 28 Oct 2018 21:44:23 -0400
Amazon long has been considered the likely winner of JEDI contract, as it is
one of the only cloud providers with the infrastructure, funds, and security
clearance necessary to meet all of the Pentagon's requirements. The
criticism is more acute because of the Pentagon's insistence on awarding
JEDI to a single bidder, rather than several companies and contractors.

Both Oracle and IBM have filed official protests with the US Government
Accountability Office, on the grounds that the DOD's decision to award the
$10 billion contract to just one company both restricts innovation and poses
a massive security risk. "JEDI turns its back on the preferences of Congress
and the administration, is a bad use of taxpayer dollars, and was written
with just one company in mind," IBM General Manager Sam Gordy said in a
statement in advance of JEDI's bid deadline.

https://www.wired.com/story/lawmakers-seek-review-pentagon-contract-thought-favor-amazon/


The customer is always right ... re: Apple iPhones

Rob Slade <rmslade@shaw.ca>
Fri, 26 Oct 2018 10:59:22 -0700
A while back, users of older Apple iPhones started making noises about their
phones being "throttled" and running slower.

Turns out they were right.  Apple had found that, for certain applications,
if the batteries were older (and possibly dying) the demands of the
application could cause the phone to simply quit, and stop working.  So an
upgrade to the operating system checked for these conditions, and, if the
battery showed signs of failing, would dial back the CPU cycles so that the
crash wouldn't happen.

Trouble is, they didn't tell people first, didn't allow any options, and
people got upset.

Now, they probably did the right thing, technically.  (Politically, it
wasn't so smart.)  And now an Italian court has decided they did the wrong
thing, and has fined them.  (They have also fined Samsung, which may not be
guilty of anything, for the same thing.)
https://nakedsecurity.sophos.com/2018/10/26/apple-and-samsung-punished-for-slowing-down-old-smartphones/ or
https://is.gd/523V2E

If this ruling stands, it's going to make deciding on upgrades and fixes a
very complicated business.  Politically.  (It was already complicated
enough, technically ...)


Fun with source code (Medium)

Gabe Goldberg <gabe@gabegold.com>
Sun, 28 Oct 2018 15:46:23 -0400
Why the NSA Called Him After Midnight and Requested His Source Code
https://medium.com/datadriveninvestor/why-the-nsa-called-me-after-midnight-and-requested-my-source-code-f7076c59ab3d


A Dark Consensus About Screens and Kids Begins to Emerge in Silicon Valley (The New York Times)

Richard Stein <rmstein@ieee.org>
Mon, 29 Oct 2018 21:53:57 +0800
https://www.nytimes.com/2018/10/26/style/phones-children-silicon-valley.html

Mental illness traced to wireless mobile device (WMD) addiction has a label:
The 'iDisorder.' See
(https://www.nytimes.com/2012/05/13/business/in-idisorder-a-look-at-mobile-device-addiction-review.html
for a book review.

Excessive mobile device usage, induced by applications that easily
captivate, is unhealthy. Children are especially susceptible to overuse.
While there's no equivalent to the US Surgeon General's "Smoking causes
cancer" warning, strictly enforced mobile device access restrictions for
adolescents constitute wise parental guidance.

The National Institutes for Health archives several studies on the
physiological effects arising from excessive mobile device usage.

"The Potential Impact of Internet and Mobile Use on Headache and Other
Somatic Symptoms in Adolescence. A Population-Based Cross-Sectional Study"
at https://www.ncbi.nlm.nih.gov/pubmed/27255862.

"Conclusion: Results highlighted the potential impact of excessive internet
and mobile use, which ranges from different types of headache to other
somatic symptoms. Further studies are needed to confirm these findings and
to determine if there is a need for promoting preventive health
interventions, especially in school setting."

"Evaluation of mobile phone addiction level and sleep quality in university
students" at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3817775/.

"Conclusion: The sleep quality worsens with increasing addiction level.  It
was concluded that referring the students with suspected addiction to
advanced healthcare facilities, performing occasional scans for early
diagnosis and informing the students about controlled mobile phone use would
be useful."


When Trump Phones Friends, the Chinese and the Russians Listen and Learn (NYTimes)

Lauren Weinstein <lauren@vortex.com>
Wed, 24 Oct 2018 16:35:22 -0700
https://www.nytimes.com/2018/10/24/us/politics/trump-phone-security.html

  When President Trump calls old friends on one of his iPhones to gossip,
  gripe or solicit their latest take on how he is doing, American
  intelligence reports indicate that Chinese spies are often listening --
  and putting to use invaluable insights into how to best work the president
  and affect administration policy, current and former American officials
  said.  Mr. Trump's aides have repeatedly warned him that his cellphone
  calls are not secure, and they have told him that Russian spies are
  routinely eavesdropping on the calls, as well. But aides say the voluble
  president, who has been pressured into using his secure White House
  landline more often these days, has still refused to give up his iPhones.
  White House officials say they can only hope he refrains from discussing
  classified information when he is on them.

So, Trump's cellphone use is being routinely monitored by our adversaries.
Perhaps part of his plan?


"Apple appears to have blocked GrayKey iPhone hacking tool" (Lucas Mearian)

Gene Wirchenko <genew@telus.net>
Tue, 30 Oct 2018 13:05:51 -0700
Lucas Mearian, Computerworld | Oct 25, 2018
Apple and two companies that have worked to enable iPhone
de-encryption continue their back-and-forth efforts.
https://www.computerworld.com/article/3268729/apple-ios/apple-appears-to-have-blocked-graykey-iphone-hacking-tool.html

selected text:

Apple has apparently been able to permanently block de-encryption technology
from a mysterious Atlanta-based company whose blackbox device was embraced
by government agencies to bypass iPhone passcodes.

Atlanta-based Grayshift is one of two companies that claimed it could thwart
Apple iPhone passcode security through brute-force attacks.

The blackbox technology purportedly worked, as Grayshift's technology was
snapped up by regional law enforcement and won contracts with Immigration
and Customs Enforcement (ICE) and the U.S. Secret Service.

All GrayShift customers sign very strict non-disclosure agreements, as any
leaked information could help Apple close the vulnerabilities they are
using, whether they find them themselves or buy zero-day flaws in Darknet,
said Vladimir Katalov, CEO of Russian forensic tech provider ElcomSoft.

"Honestly, we are not absolutely sure that the hole has been completely
closed; or maybe they will still find a workaround, or develop/buy another
way," Katalov said via email. "So that is [a] cat and mouse game that is
still ongoing. Now..., GrayShift will probably spend even more efforts to
hide their findings from the media.

"That is probably good for law enforcement, but definitely bad for the
community, as it leaves some doors still open," Katalov added.  "That's only
a question of time when GrayKey will become available to some criminals."

  [The usual about the cat and mouse game.  What I am wondering is whether
  those non-disclosure agreements are actually enforceable?]


Re: Toward Human-Understandable, Explainable AI (RISKS-30.88)

DJC <djc@resiak.org>
Thu, 25 Oct 2018 09:57:27 +0200
We're wary about giving present-day AI the power to make decisions, partly
because we don't know *why* it makes particular decisions, so its
objectivity, fairness, common sense, etc., are opaque.  At least where human
beings decide, we can ask them the basis for their decisions.

But as a matter of fact—honesty and integrity aside—humans aren't very
good at knowing the grounds for their important decisions. Daniel Kahneman
got the Nobel Prize for studying the reality of how people decide; cf. his
book "Thinking, Fast And Slow".  He and his colleagues did many, many
experiments to expose the *real* bases for how people make decisions; and
those bases are often not only unknown to their subjects, but impossible for
them to know, because they happen in inaccessible processes of their
cognition.  Yet some of those processes can be exposed through careful
experimentation over people's concrete behavior—not what they
self-report, but what they *do*.  And that was worth a Nobel Prize.

Kahneman acknowledges the impossibility of knowing everything about how one
makes one's decisions, much less controlling it all.  (In his book he
proposes some personal strategies to ameliorate how bad it can be.)

So what hope have we of transparency for the ever-more-complex AI mechanisms
into which, even already today, we have no insight at all?  Should we demand
that, at a certain level of "importance", an AI system should be subject to
the kind of concrete experimentation that Kahneman carried out in his
research?  How do we even know what to look for?

Though I'm all in favor of the kind of transparency Hani Hagras proposes, I
find it difficult to imagine how we can effectively grasp and achieve it.

I can, though, imagine that if you're planning to do something of
consequence—possibly bad consequence—that can be accomplished only
through mechanisms neither you nor anyone else can understand, it may be
time to step back and, simply, not do it.  And that notion isn't new with
AI.


Re: Explainable AI Simulation for AVs

Richard Stein <rmstein@ieee.org>
Thu, 25 Oct 2018 18:37:52 +0800
Explainable AI (XAI), per http://catless.ncl.ac.uk/Risks/30/88/%23subj3.1
posits that (T)ransparency, (C)ausality, (B)ias, (F)airness, (S)afety
characteristics must be demonstrable for an AI platform to establish a basis
for triage and public comprehension of exhibited AI behavior.

As a release metric, suppose that AV operational control program (OCP), the
vehicular equivalent of an aircraft Operational Flight Program (OFP) has to
demonstrate viability V = T + C + B + F + S == 5 (assigning 1 point for each
XAI viability factor if it passes the stimulus/response pass-fail criterion,
0 if not), and don't publish the OCP bits until it does. Publishing with a
viability score of 4, should (S)afety fail, implies significantly
compromised XAI. Potential unexplained defect escape and elevated risk of AV
OCP underachievement—meaning public safety traffic incident frequency is
likely to be higher, placing the AV's brand in jeopardy.

Note: Release viability includes additional factors that I'm not being
explicit about. Memory/descriptor leak, basic OCP function/operation,
performance, payload/message passing, built-in-self test, behavior under
sensor/processor error or fail-over conditions, etc. comprise a big
"foundational" readiness component to deterministically achieve before
attempting XAI qualification.

Given a pile of GPUs or equivalent, construct a fictitious city-scape, that
also has rural and suburban characteristics (buildings, fireplugs, houses,
bushes, trees, parks, squirrels, etc). Have people, dogs, motorcycles, and
other obstacles pop out into the driving surface, or on sidewalks at various
distances/times, at controlled intersections, randomly/unexpectedly cross
the street on bicycles, wheelchairs, scooters, skateboards, etc. Vary the
weather conditions, terrain, pavement markers, hostile WiFi DoS stimulus,
earthquakes, lighting, etc.  Conceal obstacles or scenery, and then reveal
it (remove billboards or restaurant placards), throw in some bicycles that
swerve to avoid "dooring" incidents, or even experience "dooring" and toss
out some tacoed bicycle wheels and prostrate bicyclists. Use buses,
streetcars, street sweepers, free-rolling baby trolleys, swerving vehicles,
ambulances/emergency vehicles, small aircraft landing, overturned fuel
trucks, fiddle with the sound system, a/c, power seats, windows and door
locks, sunroof, etc.

The AV simulation's stimulus must generate real-time perspective images and
sensor signaling content as detected/interpreted by LiDAR, BlueTooth, WiFi,
RADAR, or whatever comprises standard AV sensor suites. Each stimulus
condition must trace to one or more of the XAI viability attributes: T, C,
B, F, S.

Run the simulation for at least and equivalent of ~160 kilometers (100
miles) @ 60 MPH/100 KPH duration with stops, traffic jams, parallel parking,
highway merges, varying speeds, etc. and process the log files to show that
V is achieved unconditionally or with five or more nines reliability. Then
randomly modify it, and run again and repeat, for a total of ~1.6Mhours to
show V deterministically achieves or over-achieves the viability score
threshold required to publish.  Publicly release all the AV OCP simulation
stimulus conditions and processing results for review.

https://teslatap.com/undocumented/model-s-processors-count/ says a model-S
has ~65 cores among its LRUs (line replaceable units) suite. Call it 100
cores to host LRU software stacks for sensor stimulus. That implies 100
cores x 100 inputs/sec = 10000 inputs for the cores to process and output
per second. 10000 events/sec x 3600 secs/H x 1.6H = ~58M simulated sensor
stimulus inputs to generate, process, and output log for one
scenario. Assumes the AV OCPs landscape is pre-generated, save for random
physical perturbations (weather, obstacles, etc). Each scenario must be
reproducible to assist thorough triage and reconstruct anomalies that
generate a viability score less than 5. The scenario generator would be a
"work of art" unto itself.

To complete OCP qualification by divide and conquer in 1 week (24*7 = 168H)
of wall clock means ~9Kcores + memory + disk + net, etc. rigged for
real-time processing. Feed a credit card to Amazon Web Services and
provision a hunk of data center (GovCloud, r5 instances, reserved for 1
year, etc) gives ~US$ 42M for data center with 1000 GBs of network I/O.
$42M/52 weeks ~= US$ 807K/data center week.

Given this XAI simulator qualification scenario, the key question I think,
is what objective criteria are used to specify and constitute T, C, B, F,
and S for stimulus input and measurement? What standards are relevant, and
should these factors be legislated and subject to regulation by an
independent, conflict-free panel?

If there's regulatory oversight for AV OCP pre-deployment qualification,
would AV XAI be achieved under an ethically reasonable, publicly acceptable,
and sufficiently rigorous process that entitles manufacturer indemnification
against AV incidents and fatal accidents? Can any manufacturer engineer and
achieve to XAI's expected qualification rigor?

Please report problems with the web pages to the maintainer

x
Top