[extropy-chat] Amazon's Statistically Improbable Phrases

[Neil Halelamien]

On Mar 31, 2005 11:31 AM, "Hal Finney" <hal at finney.org> wrote:
> For Vinge's Fire Upon the Deep, the SIPs are: coldsleep boxes, radio
> cloaks, her dataset, his fronds, voder voice, drive spines, cargo shell,
> flying house, command deck, refugee ship, scarred one, inner keep, alien
> member, other hull, single pack, zero gee, most packs.  Seems like kind
> of weird choices: command deck? refugee ship?  Would they really be
> that rare?  And where are "Straumli perversion", or "zones of thought",
> key phrases that drove the entire structure of the book?

My guess is that the algorithm only chooses phrases which occur at
least periodically in other literature. Otherwise, you'd  just end up
with a bunch of place and people names from the book. I also suspect
it may refrain from including proper/capitalized nouns for the same
reason.

In any case, it's a very cool trick.

-- Neil