[extropy-chat] limits of computer feeling

[Anders Sandberg]

Stathis Papaioannou wrote:
> Mind modification would completely overturn the current economic system.
> Why
> would you choose brand X over brand Y if you could obtain as much reward
> from one as the other, or from neither?

They would advertise second order desires. "You *should want* to want this!"

Besides, how easy is it to set one's desires? Even if it is technically as
simple as in Greg Egan's _Reasons to be Cheerful_ it is obvious to anybody
with half a brain that there are risks involved, some of which are very
subtle. Most likely desires are very complex systems few will have the
skills to adjust well. Hence there is a market for services related to
desire management. It will range from desire consultants and designers
over desire monitors to the psycops and mind auditors who occasionally
check that you are not stuck in a bad mental state, insanity and
addiction. Compare to the role of the House AIs in Wright's golden
transcendence books. Clearly there is a huge market in a desire economy!

> And why would anyone be an advertiser or a torturer
> if
> they could get the same reward doing something fundamentally worthwhile,
> or
> doing nothing?

Isn't getting people to do something you want a wonderful challenge,
especially if they can control their own desires? It would be a grand
challenge, and some people would regard that as worthwhile.


-- 
Anders Sandberg,
Oxford Uehiro Centre for Practical Ethics
Philosophy Faculty of Oxford University