The leading search engines, such as Google, Bing and Yahoo!, use crawlers to find pages for their algorithmic search results. Pages that are linked from other search engine indexed pages do not need to be submitted because they are found automatically. The Yahoo! Directory and DMOZ, two major directories which closed in 2014 and 2017 respectively, both required manual submission and human editorial review.[40] Google offers Google Search Console, for which an XML Sitemap feed can be created and submitted for free to ensure that all pages are found, especially pages that are not discoverable by automatically following links[41] in addition to their URL submission console.[42] Yahoo! formerly operated a paid submission service that guaranteed crawling for a cost per click;[43] however, this practice was discontinued in 2009.
For some reason I had to delete some pages, these pages are using the HTML suffix, so I blocked them in robots.txt use Disallow: /*.html, but it’s been almost a year, I found that google robot often capture these pages, How can I quickly let Google completely remove these pages? And I have removed these URL from google webmaster tool by google index-> remove URLs, but Google still capture these pages.

When Googlebot crawls a page, it should see the page the same way an average user does15. For optimal rendering and indexing, always allow Googlebot access to the JavaScript, CSS, and image files used by your website. If your site's robots.txt file disallows crawling of these assets, it directly harms how well our algorithms render and index your content. This can result in suboptimal rankings.

To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]


Excellent post Brian. I think the point about writing content that appeals to influencers in spot on. Could you recommend some good, manual strategies through which I can spot influencers in boring niches *B2B* where influencers are not really talking much online? Is it a good idea to rely on newspaper articles to a feel for what a particular industry is talking about? Would love to hear your thoughts on that.
Content is king. That’s the saying, right? It’s true in a way. Your website is really just a wrapper for your content. Your content tells prospects what you do, where you do it, who you have done it for, and why someone should use your business. And if you’re smart, your content should also go beyond these obvious brochure-type elements and help your prospective customers achieve their goals.
Hey Brian, love your site + content. Really awesome stuff! I have a question about dead link building on Wikipedia. I actually got a “user talk” message from someone moderating a Wikipedia page I replaced a dead link on. They claimed that “Wikipedia uses nofollow tags” so “additions of links to Wikipedia will not alter search engine rankings.” Any thoughts here?
Thanks Brian. I’ve had a “a-ha” moment thanks to you! Great advice. I knew that backlinks would improve the organic SEO rankings to our client-targeted landing pages but I never knew it was through getting influencers to backlink blogs. I always just assumed it was great content that users wanted to share with others. It was driving me mad why people love my content but never share enough. Now I know!
Think interviews are only for the big leaguers? You’d be amazed how many people will be willing to talk to you if you just ask them. Send out emails requesting an interview to thought leaders in your industry, and publish the interviews on your blog. Not only will the name recognition boost your credibility and increase traffic to your website, the interviewee will probably share the content too, further expanding its reach.
Hey Brian, I have landed in this blog while visiting via blog land. I must appreciate your effort to put up such informative content. As being an Internet Marketing Consultant, I would like to add few thought of my own with your valuable content. There are many people who wants HUGE number of traffic with no time at all. But as per my experience, SEO has become SLOW-BUT-STEADY process in the recent times. After so many algorithm updates of Google, I think if we will do any wrong things with the websites, that should be paid off. So without taking any risk, we need to work ethically so that slowly the website will get the authority and grab the targeting traffic. What do you think mate? I am eagerly looking forward to your reply and love to see more valuable write-ups from your side. Why don’t you write about some important points about Hummingbird Updates of Google. It will be a good read. Right brother? 🙂
If you haven’t used software like BuzzSumo to check out what your competitors are up to, you’re at a huge disadvantage. These services aggregate the social performance of specific sites and content to provide you with an at-a-glance view of what topics are resonating with readers and, most importantly, making the rounds on social media. Find out what people are reading (and talking about), and emulate that kind of content to bring traffic to your website.
I am a little confused on your first point. Sorry if it is a simple one to understand and I’m just missing it. What good would finding dead links on Wiki do for my personal website? I thought you would explain how to find dead links faster within my own site… but it seems that your tip is way more valuable than that. I just don’t quite understand what I do to positively affect MY site with this. Any help would be great 🙂 THANKS!

We now have a dedicated SEO strategist who, among other things, develops 90 day plans for our websites. 90 days isn't longterm planning, but at least we have a strategic objective for the quarter. He also works closely with our UX team to identify the target audience - the crew that does the persona research and focus groups prior to the wireframe stage.
All sites have a home or "root" page, which is usually the most frequented page on the site and the starting place of navigation for many visitors. Unless your site has only a handful of pages, you should think about how visitors will go from a general page (your root page) to a page containing more specific content. Do you have enough pages around a specific topic area that it would make sense to create a page describing these related pages (for example, root page -> related topic listing -> specific topic)? Do you have hundreds of different products that need to be classified under multiple category and subcategory pages?
Use your keyword list to determine how many different pillar pages you should create. Ultimately, the number of topics for which you create pillar pages should coincide with how many different products, offerings, and locations your business has. This will make it much easier for your prospects and customers to find you in search engines no matter what keywords they use.
Page and Brin founded Google in 1998.[23] Google attracted a loyal following among the growing number of Internet users, who liked its simple design.[24] Off-page factors (such as PageRank and hyperlink analysis) were considered as well as on-page factors (such as keyword frequency, meta tags, headings, links and site structure) to enable Google to avoid the kind of manipulation seen in search engines that only considered on-page factors for their rankings. Although PageRank was more difficult to game, webmasters had already developed link building tools and schemes to influence the Inktomi search engine, and these methods proved similarly applicable to gaming PageRank. Many sites focused on exchanging, buying, and selling links, often on a massive scale. Some of these schemes, or link farms, involved the creation of thousands of sites for the sole purpose of link spamming.[25]
If you're looking to upload an image to a blog post, for example, examine the file for its file size first. If it's anywhere in megabyte (MB) territory, even just 1 MB, it's a good idea to use an image compression tool to reduce the file size before uploading it to your blog. Sites like TinyPNG make it easy to compress images in bulk, while Google's very own Squoosh has been known to shrink image file sizes to microscopic levels.

Wow I wish I had the comments you do. So you’re saying that by re-visiting and re-writing old posts garnered 111% more traffic by updating old posts? I feel like I go back from time to time to do this, mostly to keep information current. This tip makes me want to revisit all my old posts to see what could be updated. That’s a lot of low hanging fruit. Thanks for this one.
On another note, we recently went through this same process with an entire site redesign.  The executive team demanded we cut out over 75% of the pages on our site because they were useless to the visitor.  It's been 60 days since the launch of the new site and I've been able to still increase rankings, long-tail keywords, and even organic traffic.  It took a little bit of a "cowboy" mentality to get some simple things done (like using 301's instead of blocking the old content with robots.txt!).  I predicted we would lose a lot of our long tail keywords...but we haven't....yet!

Hats off to your detailing and intelligence. I thoroughly enjoyed reading the post, very informative and engaging. I was actually applying them to see the amazing results. I also found a platform called soovledotcom which actually pulls keywords from amazon, e-bay, yahoo answer, wikipedia, google and bing, but your illustrations here will certainly yeild superior results for organic seo & finding keywords.
You can confer some of your site's reputation to another site when your site links to it. Sometimes users can take advantage of this by adding links to their own site in your comment sections or message boards. Or sometimes you might mention a site in a negative way and don't want to confer any of your reputation upon it. For example, imagine that you're writing a blog post on the topic of comment spamming and you want to call out a site that recently comment spammed your blog. You want to warn others of the site, so you include the link to it in your content; however, you certainly don't want to give the site some of your reputation from your link. This would be a good time to use nofollow.
Hello Brian, really such an informative article and is more meaningful as you provided screen shots. I have noticed that articles with images bring more value to understand the things. I have just started my career in this industry and thus keep looking for some good articles/blogs that are meaningful and help me to implement tips in my work apart from my seniors instructions. I guess this was I can prove them about my caliber 🙂

I’d add one thing to number 5: Writing good copy is crucial not just for your Title/snippet, but for your whole page, especially your landing page. You want people to stay on your page for a while and (hopefully) even navigate to other pages you have. Google looks at bounce rate and where they go after they hit your page. Learning to write good copy can not only increase conversion (if you’re selling something) but make your content more impactful and engaging. There are free books at most libraries or online to help.
In the early days of the web, site owners could rank high in search engines by adding lots of search terms to web pages, whether they were relevant to the website or not. Search engines caught on and, over time, have refined their algorithms to favor high-quality content and sites. This means that SEO is now more complex than just adding the right words to your copy.
Creating high quality content takes a significant amount of at least one of the following: time, effort, expertise, and talent/skill. Content should be factually accurate, clearly written, and comprehensive. So, for example, if you describe your page as a recipe, provide a complete recipe that is easy to follow, rather than just a set of ingredients or a basic description of the dish.
There are many SEO tactics you can perform on each of your website pages to increase their rank in search engines and get more visitors. This includes producing high-quality content that your audience is searching for, and writing concise meta descriptions for your pages. The meta description appears below your URL in search results. Knowing what a page is about and what will result in a click makes users much more likely to do so. On-page SEO tactics such as these are free, but do take some time. For more help with on-page SEO, check out this blog post: Google Ranking Factors: On-Page vs Off-Page SEO.
incredible post and just what i needed! i’m actually kinda new to blogging (my first year coming around) and so far my expertise has been in copy writing/seo copy writing. however link building has become tedious for me. your talk about influencing influencers makes perfect sense, but i find it difficult for my niche. my blog site is made as “gift ideas” and holiday shoppers complete with social networks. i get shares and such from my target audience, but i find that my “influencers” (i.e etsy, red box, vat19, etc.) don’t allow dofollow links and usually can’t find suitable sources. I guess my trouble is just prospecting in general.
On one specific project, one of the SEOs on my team was brought in during the wireframe stage. T­he entire product team held SEO-specific meetings every week to go over specific recommendations, taking them very seriously, and leaning on every word our team said. We were thrilled. We were hailing their efforts, promising big wins for the relaunch, and even hyping up the launch and it’s projected SEO results in the company SEO newsletter.
As I had a teacher at school who was always really picky on how to draw conclusions I must say that the conclusions you drew for your health situation might be true, but dangerous. For example: If slightly more women than men suffer from health deseases it could be wise to write the information toward women. But, if you take search behaviour into account thing could look a lot different: It might turn up that men search more than women or that (senior) men are more present on the net than women.
Firstly, a disclaimer – don’t spam Reddit and other similar sites hoping to “hit the jackpot” of referral traffic, because it’s not going to happen. Members of communities like Reddit are extraordinarily savvy to spam disguised as legitimate links, but every now and again, it doesn’t hurt to submit links that these audiences will find genuinely useful. Choose a relevant subreddit, submit your content, then watch the traffic pour in.

A navigational page is a simple page on your site that displays the structure of your website, and usually consists of a hierarchical listing of the pages on your site. Visitors may visit this page if they are having problems finding pages on your site. While search engines will also visit this page, getting good crawl coverage of the pages on your site, it's mainly aimed at human visitors.
Google has recently changed how you can use the Google Keyword Planner. Before, everyone who signed up could see the search volume for keywords. Now, it only shows estimates. There is a way to get around this. You need to create a Google Adwords campaign. The amount you spend doesn’t matter. After you do that, you will regain access to the search volume.
It’s an awesome post which I like the most and commenting here for the first time. I’m Abhishek founder of CouponMaal want to know more like you’ve said above in the points relaunch your old posts. Here I want to know is there any difference between changing the date, time and year while we’re relaunching old post OR we should relaunch the old post with the previous date, time and year. I mean it matters or not.
Thanks so much for this entry, Laura! I loved the way your post is so practical, straightforward, newbie-friendly - and most importantly, how it emphasizes the bottom line at all times. It's easy to get "lost in the fog" of SEO with so many looming tasks and forget the main purpose, so it's wonderful to have a straightforward outline of what to do and why certain tasks need to be done. I look forward to reading your future insights!
Nice work Laura! This is going to be a great series. I'm working my way through SEOmoz's Advanced SEO Training Series (videos) Vol. 1 & 2 to build upon the advice and guidance that you and your team provided to me during my time at Yahoo!. Now many others will benefit from your knowledge, experience and passion for SEO strategy and tactics. Best wishes for great success in your new role.
Great content. Although I disagree with ‘the best times to post’ section. It is important to understand your audience. For example, if your brand/business is in high school, there will be low engagement until 2-5 when they are out of school. I highly suggest using instagram analytics (a subsidiary of facebook analytics) which gives you all of the details on when your followers are active. https://www.facebook.com/help/788388387972460
Brian, I recently found your blog by following OKDork.com. Just want to say you’re really amazing with the content you put out here. It’s so helpful, especially for someone like me who is just starting out. I’m currently writing posts for a blog I plan to launch later this year. I think my niche is a little too broad and I have to figure out how to narrow it down. I essentially want to write about my current journey of overcoming my fears to start accomplishing the dreams i have for blogging, business, and travel. In doing so, I will share the best tips, tools, and tactics I can find, as well as what worked, what didn’t and why.
Description meta tags are important because Google might use them as snippets for your pages. Note that we say "might" because Google may choose to use a relevant section of your page's visible text if it does a good job of matching up with a user's query. Adding description meta tags to each of your pages is always a good practice in case Google cannot find a good selection of text to use in the snippet. The Webmaster Central Blog has informative posts on improving snippets with better description meta tags18 and better snippets for your users19. We also have a handy Help Center article on how to create good titles and snippets20.
Having an industry influencer publish a blog post on your site or turning an interview with them into a blog post can help to drive traffic both through organic search but also via that influencer promoting the content to their audience (see the backlinks section above). This can also help to add more variety to your content and show your visitors that you are active in your field.
A user-feedback poll is one great, easy way to help you better understand your customers. Kline claims, “Done incorrectly, these can be annoying for a user. Done well, it’s an excellent opportunity to help the customer feel that their opinion matters, while also getting needed insights to better market the company. One poll we ran for an e-commerce client helped us learn that 80% of potential customers cared more about the performance of the product than the price. [So,] we added as much helpful performance information to the website as we could.”
Google recommends that all websites use https:// when possible. The hostname is where your website is hosted, commonly using the same domain name that you'd use for email. Google differentiates between the "www" and "non-www" version (for example, "www.example.com" or just "example.com"). When adding your website to Search Console, we recommend adding both http:// and https:// versions, as well as the "www" and "non-www" versions.
Thanks Brian for your article. I am in the healthy living niche. I want to team up with bloggers in my own niche where we can share material it makes sense to me. But I have my own unique message and that is what I have been devoted to! Dah! I see now that my focus should be on what is popular among my peers and add to this. I think I’m finally getting the picture! I am specifically into FOOD MEDICINE perhaps I should start writting about the dangers of a Gluten free diet! Not for everyone!
This is excellent and should be intuitive for marketers (and SEO pros are marketers!) but we often take the short cut and neglect critical details. What would also reinforce the strategy is way of providing solid projections for SEO (these could be based on industry trends and statistics). Clients now ask for ways to calculate ROI and they need numbers to get budget approvals. Increase in traffic by X,  increase in qualified traffic and leads, conversions etc, some way of quatifying the expected return.
Give customers the ways with which they can access the translated version of your website easily. And if they are not able to execute that, then they will bounce without engaging. You can integrate the ‘hreflang” attribute to the website’s code and assure that the adequately translated version of the website appears in the search engines. Yandex and Google highly recognize it.

Website owners recognized the value of a high ranking and visibility in search engine results,[6] creating an opportunity for both white hat and black hat SEO practitioners. According to industry analyst Danny Sullivan, the phrase "search engine optimization" probably came into use in 1997. Sullivan credits Bruce Clay as one of the first people to popularize the term.[7] On May 2, 2007,[8] Jason Gambert attempted to trademark the term SEO by convincing the Trademark Office in Arizona[9] that SEO is a "process" involving manipulation of keywords and not a "marketing service."

×