[extropy-chat] how does google find out everything?

[BillK]

On 8/18/06, Brian Atkins wrote:
> Also, Google and others are now providing ways for webmasters to directly
> submit site pages in order to make sure the maximum number of pages are
> indexed, and also allows you to check if there were any problems spidering your
> pages:
>
> https://www.google.com/webmasters/sitemaps/login
>
> This helps get around the need to have spider-traversable links to every page.
> --

Yes, the Google sitemaps mean you can give Google a list of all the
URLs in your site and run a check to see if the Google spiders find
any problems with scanning your site. But this doesn't necessarily
improve your Google indexing.

Google say:
A Sitemap provides an additional view into your site (just as your
home page and HTML site map do). This program does not replace our
normal methods of crawling the web. Google still searches and indexes
your sites the same way it has done in the past whether or not you use
this program.
and,
Google Sitemaps is an easy way for you to submit all your URLs to the
Google index and get detailed reports about the visibility of your
pages on Google. With Google Sitemaps, you can automatically keep us
informed of all of your current pages and of any updates you make to
those pages. Please note that submitting a Sitemap doesn't guarantee
that all pages of your site will be crawled or included in our search
results.
------------------------------

Google cannot index every web page in existence. The web is just too
big. Their spiders don't find every page and those that they do find
have to be trimmed down by their complex Page Rank system, relevance,
importance, unique information, deleting spam sites, deleting stuff
they don't approve of, etc.

What's that? You didn't know that Google censors the net?
(For your own good, of course. But don't annoy them or your site may
disappear from their search results).
See their quality guidelines at:
<http://www.google.com/support/webmasters/bin/answer.py?answer=35769>
Quote:
If a site doesn't meet our quality guidelines, it may be blocked from the index.


As an aside, the recent scare with AOL releasing customer search
details has pointed out that Google also stores all your search
details.  Use Scroogle if you want to avoid this and the Google Ads.
Clusty and ixquick also don't store search queries.


BillK