Web 2.0 Indexer

There are about 3 standard steps for any search engine to perform, such as crawling, by which content is found, indexing , where the content it’s evaluated and saved in huge databases, and retrieval, in which a user query brings a summary of relevant web pages. Crawling and indexing are processes which can take some time and which rely on many factors. Imagine if you are looking for creating a list of all the books you have, like their publisher and the amount of pages. Reviewing every book is the crawl and creating their list is the index. That’s kind of a work which search engines can do.

Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index. Google's crawl process begins with a list of web page URLs, generated from previous crawl processes, and augmented with Sitemap data provided by webmasters. As Googlebot visits each of these websites it detects links on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.

Indexing is simply the spider’s way of processing all the data from pages and sites during its crawl around the web. The spider notes new documents and changes, which are then added to the searchable index Google maintains, as long as those pages are quality content and don’t trigger alarm bells by violating Google’s user-oriented mandate. Indexing exists to ensure that users questions are promptly answered as quickly as possible. In order for your site to rank well in search results pages, it's important to make sure that Google can crawl and index your site correctly.

Sitemaps and robots.txt

Sitemaps are an important way of communication with search engines. While in robots.txt you tell search engines which parts of your site to exclude from indexing, in your sitemap you tell search engines where you'd like them to go. Sitemaps offer the opportunity to inform search engines immediately about any changes on your site. Sitemaps also help in classifying your site content, though search engines are by no means obliged to classify a page as belonging to a particular category or as matching a particular keyword only because you have told them so. There are two types of sitemaps: HTML sitemap (written in Hypertext Markup Language) and XML sitemap (written in Extensible Markup Language)

Robots.txt is a file that gives strict instructions to search engine bots about which pages they can crawl and index and which pages to stay away from. When search spiders find this file on a new domain, they read the instructions in it before doing anything else. If they don’t find a robots.txt file, the search bots assume that you want every page crawled and indexed. Duplicate content is potentially a problem for SEO. In this case you have to edit your robots.txt file and instruct search engines to ignore one of the duplicate pages. You can generate a robots.txt file quickly and easily using a Robot Control Code Generation Tool.