{"id":178,"date":"2013-09-18T07:45:59","date_gmt":"2013-09-18T14:45:59","guid":{"rendered":"https:\/\/www.evisible.co\/blog\/?p=178"},"modified":"2013-09-08T19:51:36","modified_gmt":"2013-09-09T02:51:36","slug":"dont-forget-robots-txt-files","status":"publish","type":"post","link":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/","title":{"rendered":"Don\u2019t Forget About Robots.txt Files"},"content":{"rendered":"<p>Anyone with a basic understanding of <a href=\"https:\/\/www.evisible.co\/search-engine-optimization\/\">search engine optimization<\/a> knows that meta data such as meta descriptions, image alt text and title tags is critical to proper SEO success. But there\u2019s one element that many people forget that can cause SEO campaigns to fail. Not properly implementing a Robots.txt file can make the difference between seeing your search engine rankings soar or sink.<\/p>\n<p><a href=\"https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-180\" alt=\"robots.txt\" src=\"https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt-300x189.jpg\" width=\"300\" height=\"189\" srcset=\"https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt-300x189.jpg 300w, https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt-600x378.jpg 600w, https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt-80x50.jpg 80w, https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt-30x18.jpg 30w, https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt-20x12.jpg 20w, https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt.jpg 750w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<h2>Defining a Robots.txt File<\/h2>\n<p>Simply put, a Robots.txt file can help you tell search engines which directories on your site you don\u2019t want them to index. The reasons for this are varied. You might want to make sure to keep sensitive information such as customer banking information on your eCommerce site secure. Or you might have proprietary information posted on certain sections of your website that you want to keep private. Having a Robots.txt file will tell Google, Bing and other legitimate search engines to not index these pages.<\/p>\n<p>&nbsp;<\/p>\n<h2>Robots.txt Keys for Implementation<\/h2>\n<p><b>Disable access to sensitive directories. <\/b>This can potentially include directories such as: \/cgi-bin\/, \/wp-admin\/, \/cart\/ and \/scripts\/.<\/p>\n<p><b>Remove all barriers to main content. <\/b>This includes making sure that there are no \u201cno follow\u201d tags that will block searches.<\/p>\n<p><b>Don\u2019t let search engines index \u201cduplicate\u201d pages on your website. <\/b>This can include sections of your website that are designed for regular viewing and printing, or content that is designed specifically for mobile sites. It\u2019s better to only have them index the main content page in these cases.<\/p>\n<p>&nbsp;<\/p>\n<h2>Things to Avoid<\/h2>\n<p><b>Putting comments on your Robots.txt file<\/b><\/p>\n<p><b>Listing all files in your Robots.txt. <\/b>This actually makes it easier to find files you want to keep hidden.<\/p>\n<p><b>Don\u2019t use a \/allow tag. <\/b>This doesn\u2019t exist in the Robots.txt file.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Anyone with a basic understanding of search engine optimization knows that meta data such as meta descriptions, image alt text and title tags is critical to proper SEO success. But there\u2019s one element that many people forget that can cause SEO campaigns to fail. Not properly implementing a Robots.txt file can make the difference between [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":180,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[39,7],"tags":[55,57],"class_list":["post-178","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-educational-resources","category-technical-seo-issues","tag-how-to","tag-tips"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Don\u2019t Forget About Robots.txt Files | eVisible Blog<\/title>\n<meta name=\"description\" content=\"Proper implementation of the Robots.txt file is critical for search engine optimization success. Make sure you are handling your Robots.txt files the right way.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"eVisible\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/\"},\"author\":{\"name\":\"eVisible\",\"@id\":\"https:\/\/www.evisible.co\/blog\/#\/schema\/person\/49389cda5c49ee677ca7b03bf0508653\"},\"headline\":\"Don\u2019t Forget About Robots.txt Files\",\"datePublished\":\"2013-09-18T14:45:59+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/\"},\"wordCount\":305,\"image\":{\"@id\":\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt.jpg\",\"keywords\":[\"How To\",\"Tips\"],\"articleSection\":[\"Educational Resources\",\"Technical SEO Issues\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/\",\"url\":\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/\",\"name\":\"Don\u2019t Forget About Robots.txt Files | eVisible Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.evisible.co\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt.jpg\",\"datePublished\":\"2013-09-18T14:45:59+00:00\",\"author\":{\"@id\":\"https:\/\/www.evisible.co\/blog\/#\/schema\/person\/49389cda5c49ee677ca7b03bf0508653\"},\"description\":\"Proper implementation of the Robots.txt file is critical for search engine optimization success. Make sure you are handling your Robots.txt files the right way.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#primaryimage\",\"url\":\"https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt.jpg\",\"contentUrl\":\"https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt.jpg\",\"width\":750,\"height\":473,\"caption\":\"robots.txt\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.evisible.co\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Don\u2019t Forget About Robots.txt Files\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.evisible.co\/blog\/#website\",\"url\":\"https:\/\/www.evisible.co\/blog\/\",\"name\":\"eVisible Blog\",\"description\":\"Just another WordPress site\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.evisible.co\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.evisible.co\/blog\/#\/schema\/person\/49389cda5c49ee677ca7b03bf0508653\",\"name\":\"eVisible\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.evisible.co\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/dc17a297b5f771c999fce1ac899ac0e834be38b1e8e1f4d12b5967e728d8cfea?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/dc17a297b5f771c999fce1ac899ac0e834be38b1e8e1f4d12b5967e728d8cfea?s=96&d=mm&r=g\",\"caption\":\"eVisible\"},\"description\":\"eVisible\u2019s team of online writers are experts in social media, SEO and other facets of online marketing. The eVisible team has years of experience in helping clients manage their SEO, social media and other Internet marketing campaigns and has helped countless companies increase their online presence. In addition, the team includes skilled writers who create powerful content for clients in all industries. Our team does constant research to keep up-to-date on the latest industry trends including attending key trade shows and conferences and reading the industry news sites for valuable nuggets of information.\",\"url\":\"https:\/\/www.evisible.co\/blog\/author\/evisible\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Don\u2019t Forget About Robots.txt Files | eVisible Blog","description":"Proper implementation of the Robots.txt file is critical for search engine optimization success. Make sure you are handling your Robots.txt files the right way.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/","twitter_misc":{"Written by":"eVisible","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#article","isPartOf":{"@id":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/"},"author":{"name":"eVisible","@id":"https:\/\/www.evisible.co\/blog\/#\/schema\/person\/49389cda5c49ee677ca7b03bf0508653"},"headline":"Don\u2019t Forget About Robots.txt Files","datePublished":"2013-09-18T14:45:59+00:00","mainEntityOfPage":{"@id":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/"},"wordCount":305,"image":{"@id":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#primaryimage"},"thumbnailUrl":"https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt.jpg","keywords":["How To","Tips"],"articleSection":["Educational Resources","Technical SEO Issues"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/","url":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/","name":"Don\u2019t Forget About Robots.txt Files | eVisible Blog","isPartOf":{"@id":"https:\/\/www.evisible.co\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#primaryimage"},"image":{"@id":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#primaryimage"},"thumbnailUrl":"https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt.jpg","datePublished":"2013-09-18T14:45:59+00:00","author":{"@id":"https:\/\/www.evisible.co\/blog\/#\/schema\/person\/49389cda5c49ee677ca7b03bf0508653"},"description":"Proper implementation of the Robots.txt file is critical for search engine optimization success. Make sure you are handling your Robots.txt files the right way.","breadcrumb":{"@id":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#primaryimage","url":"https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt.jpg","contentUrl":"https:\/\/www.evisible.co\/blog\/wp-content\/uploads\/2013\/09\/robots-txt.jpg","width":750,"height":473,"caption":"robots.txt"},{"@type":"BreadcrumbList","@id":"https:\/\/www.evisible.co\/blog\/dont-forget-robots-txt-files\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.evisible.co\/blog\/"},{"@type":"ListItem","position":2,"name":"Don\u2019t Forget About Robots.txt Files"}]},{"@type":"WebSite","@id":"https:\/\/www.evisible.co\/blog\/#website","url":"https:\/\/www.evisible.co\/blog\/","name":"eVisible Blog","description":"Just another WordPress site","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.evisible.co\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.evisible.co\/blog\/#\/schema\/person\/49389cda5c49ee677ca7b03bf0508653","name":"eVisible","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.evisible.co\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/dc17a297b5f771c999fce1ac899ac0e834be38b1e8e1f4d12b5967e728d8cfea?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/dc17a297b5f771c999fce1ac899ac0e834be38b1e8e1f4d12b5967e728d8cfea?s=96&d=mm&r=g","caption":"eVisible"},"description":"eVisible\u2019s team of online writers are experts in social media, SEO and other facets of online marketing. The eVisible team has years of experience in helping clients manage their SEO, social media and other Internet marketing campaigns and has helped countless companies increase their online presence. In addition, the team includes skilled writers who create powerful content for clients in all industries. Our team does constant research to keep up-to-date on the latest industry trends including attending key trade shows and conferences and reading the industry news sites for valuable nuggets of information.","url":"https:\/\/www.evisible.co\/blog\/author\/evisible\/"}]}},"_links":{"self":[{"href":"https:\/\/www.evisible.co\/blog\/wp-json\/wp\/v2\/posts\/178","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.evisible.co\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.evisible.co\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.evisible.co\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.evisible.co\/blog\/wp-json\/wp\/v2\/comments?post=178"}],"version-history":[{"count":1,"href":"https:\/\/www.evisible.co\/blog\/wp-json\/wp\/v2\/posts\/178\/revisions"}],"predecessor-version":[{"id":181,"href":"https:\/\/www.evisible.co\/blog\/wp-json\/wp\/v2\/posts\/178\/revisions\/181"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.evisible.co\/blog\/wp-json\/wp\/v2\/media\/180"}],"wp:attachment":[{"href":"https:\/\/www.evisible.co\/blog\/wp-json\/wp\/v2\/media?parent=178"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.evisible.co\/blog\/wp-json\/wp\/v2\/categories?post=178"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.evisible.co\/blog\/wp-json\/wp\/v2\/tags?post=178"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}