[ExI] Who is covering corruption in AI?

[Anders Sandberg]

On 01/11/2012 18:20, Charlie Stross wrote:
> There is a body of fiction exploring this subject. Notably "The Quantum Thief" (and sequellae -- it's going to be a trilogy) by Hannu Rajaniemi[*], and, ahem, "Rule 34" by yours truly. Also "REAMDE" by Neal Stephenson treads on the fringes of that territory.
>
> [*] First SF novel I've read where (a) quantum cryptography is a vital plot point, and (b) the author understands WTF he's talking about. (Hannu acquired a PhD in string theory before he got into writing SF.)

Second that. Just finished The Fractal Prince, very good (some nicely 
described quantum shenanigans at the start; the quantum boxing approach 
to keeping something in was new to me, despite having written about AI 
boxing myself). I have just started with Rule 34.


As for the AI thief, I think it is an underappreciated societal risk. 
Right now identity theft and stealing from credit cards still require 
some hands-on work, and this limits their effectiveness and speed. 
Suppose someone found a way of automating the process of extracting the 
money? That means that the stealing could be done on computer 
timescales, and scaled up to the total number of stolen credit card 
numbers (presumably a fairly significant fraction of all of them, given 
the statistical distribution of data losses). Also, automation means 
that rare skills can be encoded in easily copyable scripts, something 
that turned hacking from a skilled activity to the domain of 
script-kiddies and scammers. A rational thief would milk a lot of people 
for a small amount, retiring wealthy to some suitably obscure 
jurisdiction with the right kind of banks. An irrational thief would 
milk for a lot more, likely getting caught but also doing a lot more 
damage to the credibility of our credit system - and that would be 
rather bad overall. And if the exploit exists and a script-like form and 
you are a thief, you will want to use it as early as possible before 
either the exploit is closed or another, less rational thief uses it. So 
you get a race to drain as many accounts as you can.

The fact that right now people do not see this scenario as very 
plausible doesn't comfort me. Because false security is just the setting 
in which nasty exploits can get maximum effect. Yes, most of the systems 
involved in identity, credit and banking are security-minded and 
accountable... but that doesn't stop a steady stream of minor crime, and 
looking at computer security we know that sufficiently devious exploits 
get through a lot of apparent armour.

-- 
Anders Sandberg,
Future of Humanity Institute
Oxford Martin School
Faculty of Philosophy
Oxford University