{"id":517,"date":"2009-02-26T16:27:55","date_gmt":"2009-02-26T22:27:55","guid":{"rendered":"http:\/\/webii.net\/blog\/?p=517"},"modified":"2022-01-27T17:13:38","modified_gmt":"2022-01-27T23:13:38","slug":"how-to-stop-search-engines-from-indexing-certain-pages","status":"publish","type":"post","link":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/","title":{"rendered":"How To Stop Search Engines From Indexing Certain Pages"},"content":{"rendered":"

Did you know that you can stop Google from indexing certain pages on your website? You can, with a certain tool called “robots.txt”<\/p>\n

Robots.txt, as you might have already guessed, is a text file that you can put on your website that will direct programs that crawl the web<\/a> (web crawling bots) and give them certain directions. If you had a page that you didn’t want any web crawler to access, you could command web crawlers not to index it.<\/p>\n

Let’s say you were running a personal website for MC Hammer, and you had a webpage concerning his finances that you didn’t want any search engines to crawl. Here’s what you would do:<\/p>\n

1) Create a file called robots.txt in the root directory of your website. If you had www.mchammer.com<\/a> <http:\/\/www.mchammer.com><\/a>, the link would look like http:\/\/www.mchammer.com\/robots.txt<\/a><\/p>\n

2) If you were trying to block web crawler access to “seenBetterDays.html” The contents of this file will look like:<\/p>\n

User-agent: *
\nDisallow: \/seenBetterDays.html<\/p>\n

If the file was in a subdirectory, it would look like this:<\/p>\n

User-agent: *
\nDisallow: \/subdirectoryname\/seenBetterDays.html<\/p>\n

The asterisk after User-agent denotes that this rule applies to all robots, not just Google, Yahoo, or any one robot specifically.
\nIf you wanted to disclude the entire subdirectory, it would look like this.<\/p>\n

User-agent: *
\nDisallow: \/subdirectoryname\/<\/em><\/p>\n

This will block web crawler access to all files within \/subdirectory name\/. If you wanted to disallow access to the entire subdirectory, except for the file “exception.html,”, you would put this:<\/p>\n

User-agent: *
\nDisallow: \/subdirectory name\/<\/em>
\nAllow: \/subdirectory name\/exception.html<\/p>\n

Finally, if you decided that you’ve had enough of the internet and all its pervasive indexing and searching, you would put in this content:<\/p>\n

User-agent: *
\nDisallow: \/<\/p>\n

This basically means no robot will ever visit www.mchammer.com<\/a> <http:\/\/www.mchammer.com><\/a> again until you remove the file.<\/p>\n

There are many, many, many more uses for robots.txt, but we’ve covered the basics in this article so that if you have a certain webpage or group of webpages that you don’t want to be crawled, you can just include a file and keep your information a little bit more private.<\/p>\n","protected":false},"excerpt":{"rendered":"

Did you know that you can stop Google from indexing certain pages on your website? You can, with a certain tool called “robots.txt” Robots.txt, as you might have already guessed, is a text file that you can put on your website that will direct programs that crawl the web (web crawling bots) and give them […]<\/p>\n","protected":false},"author":2,"featured_media":10251,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[16,47,300],"tags":[242,55,243],"class_list":["post-517","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-austin-web-design","category-how-to","category-www-learning-center","tag-austin-web-design","tag-robotstxt","tag-seo"],"acf":[],"yoast_head":"\nHow To Stop Search Engines From Indexing Certain Pages<\/title>\n<meta name=\"description\" content=\"How do you use the robots.txt tool? We have the answer! Come read these easy steps to keeping robots off your site!\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How To Stop Search Engines From Indexing Certain Pages\" \/>\n<meta property=\"og:description\" content=\"How do you use the robots.txt tool? We have the answer! Come read these easy steps to keeping robots off your site!\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/\" \/>\n<meta property=\"og:site_name\" content=\"Web Marketing & Web Development Company Blog - WEBii\" \/>\n<meta property=\"article:published_time\" content=\"2009-02-26T22:27:55+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-01-27T23:13:38+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.webii.net\/blog\/wp-content\/uploads\/2009\/02\/Person-Coding-Image.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"640\" \/>\n\t<meta property=\"og:image:height\" content=\"423\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Bobby Martinez\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bobby Martinez\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/\"},\"author\":{\"name\":\"Bobby Martinez\",\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/#\\\/schema\\\/person\\\/1bc298cae5d352e2b7409f7f0882c776\"},\"headline\":\"How To Stop Search Engines From Indexing Certain Pages\",\"datePublished\":\"2009-02-26T22:27:55+00:00\",\"dateModified\":\"2022-01-27T23:13:38+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/\"},\"wordCount\":387,\"publisher\":{\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/wp-content\\\/uploads\\\/2009\\\/02\\\/Person-Coding-Image.jpg\",\"keywords\":[\"Austin Web Design\",\"robots.txt\",\"SEO\"],\"articleSection\":[\"Austin Web Design\",\"How To\",\"WWW Learning Center\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/\",\"url\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/\",\"name\":\"How To Stop Search Engines From Indexing Certain Pages\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/wp-content\\\/uploads\\\/2009\\\/02\\\/Person-Coding-Image.jpg\",\"datePublished\":\"2009-02-26T22:27:55+00:00\",\"dateModified\":\"2022-01-27T23:13:38+00:00\",\"description\":\"How do you use the robots.txt tool? We have the answer! Come read these easy steps to keeping robots off your site!\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/wp-content\\\/uploads\\\/2009\\\/02\\\/Person-Coding-Image.jpg\",\"contentUrl\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/wp-content\\\/uploads\\\/2009\\\/02\\\/Person-Coding-Image.jpg\",\"width\":640,\"height\":423,\"caption\":\"Person Coding Image\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/how-to-stop-search-engines-from-indexing-certain-pages\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Austin Web Design\",\"item\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/category\\\/austin-web-design\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"How To Stop Search Engines From Indexing Certain Pages\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/\",\"name\":\"Web Marketing & Web Development Company Blog - WEBii\",\"description\":\"web design . web development . seo . marketing . web hosting . + more\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/#organization\",\"name\":\"WEBii\",\"url\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/wp-content\\\/uploads\\\/2015\\\/12\\\/fb-banner-webii-seo.jpg\",\"contentUrl\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/wp-content\\\/uploads\\\/2015\\\/12\\\/fb-banner-webii-seo.jpg\",\"width\":1200,\"height\":598,\"caption\":\"WEBii\"},\"image\":{\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/#\\\/schema\\\/person\\\/1bc298cae5d352e2b7409f7f0882c776\",\"name\":\"Bobby Martinez\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/wp-content\\\/wphb-cache\\\/gravatar\\\/989\\\/989ff15d4b427e880bb1f76de01c47c3x96.jpg\",\"url\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/wp-content\\\/wphb-cache\\\/gravatar\\\/989\\\/989ff15d4b427e880bb1f76de01c47c3x96.jpg\",\"contentUrl\":\"https:\\\/\\\/www.webii.net\\\/blog\\\/wp-content\\\/wphb-cache\\\/gravatar\\\/989\\\/989ff15d4b427e880bb1f76de01c47c3x96.jpg\",\"caption\":\"Bobby Martinez\"},\"description\":\"Customer Support, WEBii\",\"sameAs\":[\"https:\\\/\\\/www.webii.net\"]}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How To Stop Search Engines From Indexing Certain Pages","description":"How do you use the robots.txt tool? We have the answer! Come read these easy steps to keeping robots off your site!","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/","og_locale":"en_US","og_type":"article","og_title":"How To Stop Search Engines From Indexing Certain Pages","og_description":"How do you use the robots.txt tool? We have the answer! Come read these easy steps to keeping robots off your site!","og_url":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/","og_site_name":"Web Marketing & Web Development Company Blog - WEBii","article_published_time":"2009-02-26T22:27:55+00:00","article_modified_time":"2022-01-27T23:13:38+00:00","og_image":[{"width":640,"height":423,"url":"https:\/\/www.webii.net\/blog\/wp-content\/uploads\/2009\/02\/Person-Coding-Image.jpg","type":"image\/jpeg"}],"author":"Bobby Martinez","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Bobby Martinez","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/#article","isPartOf":{"@id":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/"},"author":{"name":"Bobby Martinez","@id":"https:\/\/www.webii.net\/blog\/#\/schema\/person\/1bc298cae5d352e2b7409f7f0882c776"},"headline":"How To Stop Search Engines From Indexing Certain Pages","datePublished":"2009-02-26T22:27:55+00:00","dateModified":"2022-01-27T23:13:38+00:00","mainEntityOfPage":{"@id":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/"},"wordCount":387,"publisher":{"@id":"https:\/\/www.webii.net\/blog\/#organization"},"image":{"@id":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/#primaryimage"},"thumbnailUrl":"https:\/\/www.webii.net\/blog\/wp-content\/uploads\/2009\/02\/Person-Coding-Image.jpg","keywords":["Austin Web Design","robots.txt","SEO"],"articleSection":["Austin Web Design","How To","WWW Learning Center"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/","url":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/","name":"How To Stop Search Engines From Indexing Certain Pages","isPartOf":{"@id":"https:\/\/www.webii.net\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/#primaryimage"},"image":{"@id":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/#primaryimage"},"thumbnailUrl":"https:\/\/www.webii.net\/blog\/wp-content\/uploads\/2009\/02\/Person-Coding-Image.jpg","datePublished":"2009-02-26T22:27:55+00:00","dateModified":"2022-01-27T23:13:38+00:00","description":"How do you use the robots.txt tool? We have the answer! Come read these easy steps to keeping robots off your site!","breadcrumb":{"@id":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/#primaryimage","url":"https:\/\/www.webii.net\/blog\/wp-content\/uploads\/2009\/02\/Person-Coding-Image.jpg","contentUrl":"https:\/\/www.webii.net\/blog\/wp-content\/uploads\/2009\/02\/Person-Coding-Image.jpg","width":640,"height":423,"caption":"Person Coding Image"},{"@type":"BreadcrumbList","@id":"https:\/\/www.webii.net\/blog\/how-to-stop-search-engines-from-indexing-certain-pages\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.webii.net\/blog\/"},{"@type":"ListItem","position":2,"name":"Austin Web Design","item":"https:\/\/www.webii.net\/blog\/category\/austin-web-design\/"},{"@type":"ListItem","position":3,"name":"How To Stop Search Engines From Indexing Certain Pages"}]},{"@type":"WebSite","@id":"https:\/\/www.webii.net\/blog\/#website","url":"https:\/\/www.webii.net\/blog\/","name":"Web Marketing & Web Development Company Blog - WEBii","description":"web design . web development . seo . marketing . web hosting . + more","publisher":{"@id":"https:\/\/www.webii.net\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.webii.net\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.webii.net\/blog\/#organization","name":"WEBii","url":"https:\/\/www.webii.net\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.webii.net\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.webii.net\/blog\/wp-content\/uploads\/2015\/12\/fb-banner-webii-seo.jpg","contentUrl":"https:\/\/www.webii.net\/blog\/wp-content\/uploads\/2015\/12\/fb-banner-webii-seo.jpg","width":1200,"height":598,"caption":"WEBii"},"image":{"@id":"https:\/\/www.webii.net\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.webii.net\/blog\/#\/schema\/person\/1bc298cae5d352e2b7409f7f0882c776","name":"Bobby Martinez","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.webii.net\/blog\/wp-content\/wphb-cache\/gravatar\/989\/989ff15d4b427e880bb1f76de01c47c3x96.jpg","url":"https:\/\/www.webii.net\/blog\/wp-content\/wphb-cache\/gravatar\/989\/989ff15d4b427e880bb1f76de01c47c3x96.jpg","contentUrl":"https:\/\/www.webii.net\/blog\/wp-content\/wphb-cache\/gravatar\/989\/989ff15d4b427e880bb1f76de01c47c3x96.jpg","caption":"Bobby Martinez"},"description":"Customer Support, WEBii","sameAs":["https:\/\/www.webii.net"]}]}},"_links":{"self":[{"href":"https:\/\/www.webii.net\/blog\/wp-json\/wp\/v2\/posts\/517","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.webii.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webii.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webii.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webii.net\/blog\/wp-json\/wp\/v2\/comments?post=517"}],"version-history":[{"count":0,"href":"https:\/\/www.webii.net\/blog\/wp-json\/wp\/v2\/posts\/517\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.webii.net\/blog\/wp-json\/wp\/v2\/media\/10251"}],"wp:attachment":[{"href":"https:\/\/www.webii.net\/blog\/wp-json\/wp\/v2\/media?parent=517"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webii.net\/blog\/wp-json\/wp\/v2\/categories?post=517"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webii.net\/blog\/wp-json\/wp\/v2\/tags?post=517"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}