nettime's_risk_manager on Thu, 1 Oct 2015 17:16:19 +0200 (CEST)


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

<nettime> Risks-Forum Digest Tue 29 Sep 2015 28.97


< http://catless.ncl.ac.uk/Risks/28.97.html >

RISKS-LIST: Risks-Forum Digest  Tuesday 29 September 2015  Volume 28 : Issue 97 

ACM FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS (comp.risks) 
Peter G. Neumann, moderator, chmn ACM Committee on Computers and Public Policy 

***** See last item for further information, disclaimers, caveats, etc. ***** 
This issue is archived at <http://www.risks.org> as 
  <http://catless.ncl.ac.uk/Risks/28.97.html> 
The current issue can be found at 
  <http://www.csl.sri.com/users/risko/risks.txt> 

  Contents: 
EPA v VW cheatware, AI & "machine learning" (Henry Baker) 
Volkswagen Law (Bloomberg) 
Security Standards: cars, voting, medical, critical infrastructure, etc. 
  (Alister Wm Macintyre) 
Re: VW Scandal (Robert Schaefer) 
Gaming security (Michael Albaugh) 
Storing secret crypto keys in the Amazon cloud? New attack can steal them 
  (Ars Technica) 
GCHQ operation "Karma Police" (Slashdot) 
Network scientists have discovered how social networks can create 
  the illusion that something is common when it is actually rare (MIT) 
Law Enforcement's Love/Hate Relationship with Cloud Auto Backup 
  (Lauren Weinstein) 
Hello Barbie (The Week) 
Re: U.S. and China cyber establish 'hotline' (Henry Baker) 
Re: Ad-blocking (L. Mark Stone) 
Abridged info on RISKS (comp.risks) 

---------------------------------------------------------------------- 

Date: Sat, 26 Sep 2015 06:46:36 -0700 
From: Henry Baker <hba...@pipeline.com> 
Subject: EPA v VW cheatware, AI & "machine learning" 

The tech world is very excited, but also frightened, about AI & machine 
learning these days; we worry about AI/machine learning algorithms replacing 
doctors, lawyers, teachers, taxi drivers. 

Perhaps one of the most straightforward applications of AI & machine 
learning today would be a computer that "learns" how to control the 
emissions of a vehicle engine so that it can pass the EPA emissions tests. 

Consider the following conceptual model: a computer with a bunch (hundreds?) 
of sensors and a bunch of actuators (tens?) that watches over a diesel 
engine while it is being driven through a standard EPA emissions test. 

The computer can sense perhaps air temperature, humidity, engine speed, 
engine load, engine temperature, etc., and can control perhaps the air flow, 
the fuel flow, the flow of Adblue (aka DEF/ISO 22241), etc.  Sensors don't 
cost very much, so there may also be sensors for the engine hood/bonnet 
being open, the position of the steering wheel, etc. 

We now put this system through hundreds of thousand of miles of "learning" 
(millions of miles if the testing & learning can be virtualized & run in 
parallel), so that the AI/machine learning algorithm learns to optimize 
inputs like fuel and Adblue while still meeting EPA testing limits. 

I can guarantee you that this AI/machine learning algorithm will quickly 
notice that the best way to optimize for the EPA test is to "cheat" -- i.e., 
to notice that when the hood/bonnet is open and the steering wheel is 
straight ahead, this would be a good time to optimize NOx and other 
emissions, while during other conditions -- hood/bonnet closed and steering 
wheel twisting back & forth (perhaps a curving country lane) -- emissions 
aren't so important relative to performance. 

(Perhaps someone -- Google might be in the best position with their work on 
autonomous robots and its expertise in "machine learning" for their 
self-driving cars -- is already working on such AI/machine learning 
experiments for engine optimization; I'd be interested in hearing about them 
if anyone can send me links.) 

So is this AI/machine learning program "unethical" wrt the EPA tests ? 
Should it be fined or go to jail ? 

This is no longer idle speculation, as these AI/machine learning programs 
are "recognizing" speech, gaits, faces, writing styles, etc.  Are they also 
"cheating" ? 

As automobiles become more complex, and as machine learning algorithms 
become more sophisticated, engine optimization computers may no longer be 
"programmed" by humans using coding techniques, but will be "taught" by 
following a long sequence of example situations and "learning" the correct 
responses. 

The DMCA may no longer be relevant to such computers, because *there is no 
source code* to look at, and indeed, the *binary code* may itself simply be 
a huge pile of random-looking floating point numbers in a *neural network*. 
The *only* way to check such a system will be through exhaustive (!) 
behavioral testing, as there won't be any source code to logically check for 
"cheats" and "defeats". 

I'm not trying to excuse the VW management that has already admitted to 
"cheating" on the EPA tests, but as a computer scientist, I'm not so sure 
where we go forward from here.  We have terrific new opportunities with 
electric and self-driving cars, so "optimizing" the government regulation of 
diesel engines may simply be re-arranging the deck chairs on the Titanic. 

------------------------------ 

Date: Mon, 28 Sep 2015 18:39:20 -0500 
From: "Alister Wm Macintyre \(Wow\)" <macwh...@wowway.com> 
Subject: Volkswagen Law 

Bloomberg Business Week, 28 Sep-4 Oct issue: 

Germany has a law, about state ownership of corporations, called the 
Volkswagen Law.  Because of it, Lower Saxony owns 20% of VW, giving its 
Prime Minister virtual veto power over VW.  Thanks to the VW law, which the 
EU has been fighting, many German companies have two boards of directors. 
There is the management board, with Executives, which answers to the 
Supervisory Board, run by the state, labor leaders, and shareholders. 

As yet, there is no evidence which part of VW leadership structure had 
anything to do with the emissions deception. 

The article does not address the car hacking cover-up. 

------------------------------ 

Date: Mon, 28 Sep 2015 14:05:05 -0500 
From: "Alister Wm Macintyre \(Wow\)" <macwh...@wowway.com> 
Subject: Security Standards: cars, voting, medical, critical 
  infrastructure, etc. 

I retired early this year.  Previously I managed an IBM midrange system at 
my day job. 

The IBM OS tracked all sorts of updates, changes into a log, which we could 
examine for suspicious activity, and I regularly checked it for potential 
breaches, and some types of human error. We had some control over how much 
activity to log, and for how many days, because of disk space constraints, 
but some things, such as changes to the OS itself, we could not change the # 
days below a certain point.  I had occasion to alter the system date, so I 
learned it was theoretically possible to erase the security log by advancing 
the date, so that all contents were now past the erase date.  It was 
impossible to erase the fact that someone had messed with the system date. 

For an OS log to be meaningful, some humans need to be able to dig out 
what's important from the huge volume of frequently geeky entries. 

All the passwords were changeable, but IBM knew how to bust their own 
security.  We found this out on the occasion of a relocation, where IBM 
techs could not reassemble everything correctly, could not get into some 
diagnostics, because of the security, which was on the hard drive which was 
not reconnecting properly.  They needed authorization from our CEO to bust 
the security.  Fortunately the CEO was watching the operation, so there was 
no delay getting this. 

If this is true across vendors, then there is a potential risk from former 
and present employees of the computer vendors. 

In my IBM Midrange world, there's a software monitoring package marketed 
under the name "Needle in Haystack" which sends alerts to IT people, when 
stuff happens which can have adverse impact on the enterprise.  If the IT 
people do not respond, in a reasonable time frame, the alerts go up the 
management ladder. 

I imagine that other platforms, outside the IBM world, ought to have similar 
standards.  Capture activity of a potentially suspicious nature, make it 
available to relevant people in an intelligible and timely manner. 

The next question is who this data belongs to, who may access it, update it 
- employees, regulators, vendors of the hardware & software.  I recently had 
something weird happen with my auto, so I was re-reading the owner's manual. 
I encountered a statement that data is captured about the vehicle 
operations, and that this data belongs to the vehicle's owner. 

------------------------------ 

Date: Mon, 28 Sep 2015 15:56:11 -0400 
From: Robert Schaefer <r...@haystack.mit.edu> 
Subject: Re: VW Scandal 

For those who don't have long memories, in the late 1980s, there was serious 
attention to processor benchmarking by running compilers on well known 
libraries and benchmark test programs' source code.  It was claimed that 
certain vendor's compilers (back then, compilers were often bespoke) would 
recognize the library or test program, and optimize the number crunching to 
the point where the benchmark test became worthless.  Googling the terms 
computer+benchmark+cheating shows that this is still going on today. 

------------------------------ 

Date: Mon, 28 Sep 2015 10:37:12 -0700 
From: Michael Albaugh <m.e.a...@gmail.com> 
Subject: Gaming security (Re: Murray comment on casino slots) 

You might want to keep in mind the incident (possibly multiple) of slot 
machines gaffed by the gaming board inspectors themselves.  Back in the 
1990s, but can we be sure they do not continue? 

  [Of course not.  The gambling machines are held to a `higher' standard, 
  but it is still not a very high one.  Remember, the best is the enemy 
  of the good, but the so-called good is nowhere near good enough.  PGN] 

------------------------------ 

Date: Mon, 28 Sep 2015 12:04:00 -0700 
From: Lauren Weinstein <lau...@vortex.com> 
Subject: Storing secret crypto keys in the Amazon cloud? New attack can 
  steal them 

http://arstechnica.com/security/2015/09/storing-secret-crypto-keys-in-the-amazon-cloud-new-attack-can-steal-them/ 

  Now a separate team of researchers has constructed a new method for 
  recovering the full private key used in a modern implementation of the 
  widely used RSA crypto system. Like the 2009 work, the new research 
  implements a CPU cache attack across two Amazon accounts that happen to be 
  located on the same chip or chipset. They recently used their technique to 
  allow one Amazon instance to recover the entire 2048-bit RSA key used by a 
  separate instance, which they also happened to control. The newer 
  technique works by probing the last level cache of the Intel Xeon 
  processor chipsets used by Amazon computers. 

------------------------------ 

Date: Tue, 29 Sep 2015 12:58:41 +0200 
From: Werner U <wer...@gmail.com> 
Subject: GCHQ operation "Karma Police" (Slashdot) 

Ars Technica's story on the revelations reported today by The Intercept that 
the UK's GCHQ has been tracking World Wide Web users since 2007 with an 
operation called "Karma Police" -- "a program that tracked Web browsing 
habits of people around the globe in what the agency itself billed as the 
'world's biggest' Internet data-mining operation, intended to eventually 
track 'every visible user on the Internet.'" 
http://yro.slashdot.org/story/15/09/25/2349201/gchq-tried-to-track-web-visits-of-every-visible-user-on-internet 
<https://theintercept.com/2015/09/25/gchq-radio-porn-spies-track-web-users-online-identities/> 
<http://arstechnica.com/security/2015/09/gchq-tried-to-track-web-visits-of-every-visible-user-on-internet/>, 

------------------------------ 

Date: Mon, 28 Sep 2015 12:00:47 -0700 
From: Lauren Weinstein <lau...@vortex.com> 
Subject: Network scientists have discovered how social networks can create 
         the illusion that something is common when it is actually rare 

MIT via NNSquad 
http://www.technologyreview.com/view/538866/the-social-network-illusion-that-tricks-your-mind/ 

  Today, we get an insight into why this happens thanks to the work of 
  Kristina Lerman and pals at the University of Southern California. These 
  people have discovered an extraordinary illusion associated with social 
  networks which can play tricks on the mind and explain everything from why 
  some ideas become popular quickly to how risky or antisocial behavior can 
  spread so easily. 

------------------------------ 

Date: Mon, 28 Sep 2015 20:16:15 -0700 
From: Lauren Weinstein <lau...@vortex.com> 
Subject: Law Enforcement's Love/Hate Relationship with Cloud Auto Backup 

             http://lauren.vortex.com/archive/001126.html 

There's a story going around today regarding an individual who was arrested 
and charged with assaulting a police officer when authorities arrived over a 
noise complaint. But cellphone video recorded by the arrestee convinced a 
judge that police had assaulted him, not the other way around. What's 
particularly unusual in this case is that the arrestee's cellphone had 
"mysteriously" vanished at the police station before any video was 
discovered. 

 <...>

------------------------------ 

Date: Mon, 28 Sep 2015 18:39:20 -0500 
From: "Alister Wm Macintyre \(Wow\)" <macwh...@wowway.com> 
Subject: Hello Barbie (The Week) 

Toymaker Mattel is coming out with $75.00 Hello Barbie, a wi-fi enabled 
doll.  The little-girl owners shall press the belt buckle, and she'll ask 
questions, like "Where do you live?" and answer the child's questions.  I am 
only guessing at what questions might be asked, as the list probably not yet 
published where I can find it. The child's conversations with Hello Barbie 
will be stored on Toy Talk servers, allegedly only to help Mattel improve 
their speech recognition software.  But how long before Mattel sells this 
info to advertisers, or their data gets hacked?  Can Hello Barbie get 
software updates, so she can promote future Mattel toys, more Barbie 
clothing, etc.? 

------------------------------ 

Date: Mon, 28 Sep 2015 11:22:39 -0700 
From: Henry Baker <hba...@pipeline.com> 
Subject: Re: U.S. and China cyber establish 'hotline' (RISKS-28.96) 

Any bets about how long it will take for someone to hack this cyber 'red 
phone' / 'red skype'? 

------------------------------ 

Date: Mon, 28 Sep 2015 20:55:00 +0000 (UTC) 
From: "L. Mark Stone" <lms...@lmstone.com> 
Subject:  Re: Ad-blocking (Ross, RISKS-28.96) 

Mr. Ross's remarks regarding ad-blocking software in Risks 28.94 ends with 
the two questions: 

"Why can I not choose to block advertisements on the Internet? What is it 
about the Internet that mandates its advertisements on me, something other 
media cannot do?" 

Intended to highlight that the equivalent of Internet ad-blocking is allowed 
with radio, television, magazines and newspapers by turning the page, 
hitting the mute button or changing the channel, unfortunately the real 
answer to Mr. Ross's questions is "money". 

A quick Google search shows that Internet advertising is the largest single 
category of advertising, outpacing even television. Online advertising can 
be targeted as we know with much greater specificity than any other 
advertising outlet. Neither advertisers nor online media therefore have any 
incentive (and indeed extremely strong disincentives) to allow ad blocking 
if they have any say-so. So why would they? 

------------------------------ 

Date: Mon, 17 Nov 2014 11:11:11 -0800 
From: RISKS-...@csl.sri.com 
Subject: Abridged info on RISKS (comp.risks) 

 The ACM RISKS Forum is a MODERATED digest. Its Usenet manifestation is 
 comp.risks, the feed for which is donated by panix.com as of June 2011. 
=> SUBSCRIPTIONS: PLEASE read RISKS as a newsgroup (comp.risks or equivalent) 
 if possible and convenient for you.  The mailman Web interface can 
 be used directly to subscribe and unsubscribe: 
   http://mls.csl.sri.com/mailman/listinfo/risks 
 Alternatively, to subscribe or unsubscribe via e-mail to mailman 
 your FROM: address, send a message to 
   risks-...@csl.sri.com 
 containing only the one-word text subscribe or unsubscribe.  You may 
 also specify a different receiving address: subscribe address= ... . 
 You may short-circuit that process by sending directly to either 
   risks-s...@csl.sri.com or risks-un...@csl.sri.com 
 depending on which action is to be taken. 

 Subscription and unsubscription requests require that you reply to a 
 confirmation message sent to the subscribing mail address.  Instructions 
 are included in the confirmation message.  Each issue of RISKS that you 
 receive contains information on how to post, unsubscribe, etc. 

=> The complete INFO file (submissions, default disclaimers, archive sites, 
 copyright policy, etc.) is online. 
   <http://www.CSL.sri.com/risksinfo.html> 
 *** Contributors are assumed to have read the full info file for guidelines. 

=> .UK users may contact <Lindsay....@newcastle.ac.uk>. 
=> SPAM challenge-responses will not be honored.  Instead, use an alternative 
 address from which you NEVER send mail! 
=> SUBMISSIONS: to ri...@CSL.sri.com with meaningful SUBJECT: line. 
 *** NOTE: Including the string `notsp' at the beginning or end of the subject 
 *** line will be very helpful in separating real contributions from spam. 
 *** This attention-string may change, so watch this space now and then. 
=> ARCHIVES: ftp://ftp.sri.com/risks for current volume 
     or ftp://ftp.sri.com/VL/risks for previous VoLume 
 http://www.risks.org takes you to Lindsay Marshall's searchable archive at 
 newcastle: http://catless.ncl.ac.uk/Risks/VL.IS.html gets you VoLume, ISsue. 
   Lindsay has also added to the Newcastle catless site a palmtop version 
   of the most recent RISKS issue and a WAP version that works for many but 
   not all telephones: http://catless.ncl.ac.uk/w/r 
 <http://the.wiretapped.net/security/info/textfiles/risks-digest/> . 
==> PGN's comprehensive historical Illustrative Risks summary of one liners: 
    <http://www.csl.sri.com/illustrative.html> for browsing, 
    <http://www.csl.sri.com/illustrative.pdf> or .ps for printing 
  is no longer maintained up-to-date except for recent election problems. 
 *** NOTE: If a cited URL fails, we do not try to update them.  Try 
  browsing on the keywords in the subject line or cited article leads. 
==> Special Offer to Join ACM for readers of the ACM RISKS Forum: 
    <http://www.acm.org/joinacm1> 

------------------------------ 

End of RISKS-FORUM Digest 28.97 
************************ 


#  distributed via <nettime>: no commercial use without permission
#  <nettime>  is a moderated mailing list for net criticism,
#  collaborative text filtering and cultural politics of the nets
#  more info: http://mx.kein.org/mailman/listinfo/nettime-l
#  archive: http://www.nettime.org contact: nettime@kein.org