The following terms and conditions govern all use of the MyThemeShop.com website (“Service”) and it’s sub-domains. The Service is owned and operated by MyThemeShop LLC. (“MyThemeShop”, “MTS”, “Us, “We”, or “Our”). By using the Service, you (“You”, “Yourself” or “Your”) agree to these terms of use in full. If You disagree with these terms of use, or any part of these terms of use, You must not use the Service.

Search engine spiders can only spider through text. They will use the content on your site to determine what your site is about, which in turn will help to decide how highly your site will be ranked for specific keyword phrases when visitors type them into the search engines. For this reason, keyword research is critical to obtaining natural search engine placement and should be at the top of your list when mapping out your SEO strategy.


To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]
Nice work Laura! This is going to be a great series. I'm working my way through SEOmoz's Advanced SEO Training Series (videos) Vol. 1 & 2 to build upon the advice and guidance that you and your team provided to me during my time at Yahoo!. Now many others will benefit from your knowledge, experience and passion for SEO strategy and tactics. Best wishes for great success in your new role.
×