{"id":10582,"date":"2023-08-13T20:02:59","date_gmt":"2023-08-13T18:02:59","guid":{"rendered":"https:\/\/wajari.com\/blog\/meta-robots\/"},"modified":"2023-08-13T20:33:51","modified_gmt":"2023-08-13T18:33:51","slug":"meta-robots","status":"publish","type":"post","link":"https:\/\/wajari.com\/en\/blog\/meta-robots\/","title":{"rendered":"Meta robots"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Tabla de contenidos<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/wajari.com\/en\/blog\/meta-robots\/#Tag_syntax\" >Tag syntax<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/wajari.com\/en\/blog\/meta-robots\/#noindex_follow\" >noindex, follow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/wajari.com\/en\/blog\/meta-robots\/#index_nofollow\" >index, nofollow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/wajari.com\/en\/blog\/meta-robots\/#noindex_nofollow\" >noindex, nofollow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/wajari.com\/en\/blog\/meta-robots\/#Index_follow\" >Index, follow<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/wajari.com\/en\/blog\/meta-robots\/#Is_there_a_difference_between_robotstxt_and_meta_robots_at_the_crawling_level\" >Is there a difference between robots.txt and meta robots at the crawling level?<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/wajari.com\/en\/blog\/meta-robots\/#Other_directives_for_meta_robots\" >Other directives for meta robots<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/wajari.com\/en\/blog\/meta-robots\/#Canonical_label\" >Canonical label<\/a><\/li><\/ul><\/nav><\/div>\n\n<p>What are meta robots in SEO? First episode of a series of posts and videos that I will organize in the category of: <a href=\"https:\/\/wajari.com\/en\/categoria\/guides\/\">SEO Guides<\/a>. Basic manuals to go deeper.  <\/p>\n\n<p>In the <strong>headers of the web<\/strong>, we usually put very relevant information for search engines, invisible to the user. Meta robots are a tag in HTML that gives an instruction to search engines.<\/p>\n\n<p>This is usually the best way to control the behavior of each URL.<\/p>\n\n<p>Meta robots are a classic of <strong>technical SEO<\/strong>, one of the first things to learn when you start in this world.<\/p>\n\n<p>It is a very simple and basic subject but we must be very clear about the concepts if we do not want to make mistakes. Directly by your actions on the web or by any modification you make in any of the <strong>SEO plugins<\/strong> that we usually install in WordPress.<\/p>\n\n<p>As <a href=\"https:\/\/www.humanlevel.com\/diccionario-marketing-online\/meta-robots\" rel=\"noopener\">Fernando Maci\u00e1<\/a> points out in his digital marketing dictionary on his Human Level website:<\/p>\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>&#8220;Meta robots allows you to control how a page should be indexed and how it is displayed to users on the search results page.&#8221;<\/p>\n\n\n\n<p>Fernando Macia<\/p>\n<\/blockquote>\n\n<p>It doesn&#8217;t get any clearer than that.<\/p>\n\n<p>Also, as you will see, in <em>robots.txt<\/em> we block a URL completely, while with <strong>meta robots <\/strong>we can have a URL that is still passing link juice or popularity, but that we decide not to show up in Google&#8217;s indexes.<\/p>\n\n<div style=\"height:30px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Meta robots en SEO\" width=\"1200\" height=\"675\" src=\"https:\/\/www.youtube.com\/embed\/EnpyglY_Dzc?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n<div style=\"height:30px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Tag_syntax\"><\/span>Tag syntax<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n<p>Very simple: and these are the options we can define:<\/p>\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"noindex_follow\"><\/span>noindex, follow<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n<pre class=\"wp-block-code\"><code>&lt;meta name=\"robots\" content=\"noindex, follow\"\/&gt;<\/code><\/pre>\n\n<p>In this case with the noindex we tell search engines NOT to index this content but you canfollow the links.<\/p>\n\n<p>By following the links we maintain the link transfer and associated popularity juice.<\/p>\n\n<p>This is the most typical solution when you want to avoid indexing a URL that may be considered as thin content or duplicate content from other sections of your website.<\/p>\n\n<p>Very common in search results, which generates a change in the URL with the search term. In label files, author, etc.<\/p>\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"index_nofollow\"><\/span>index, nofollow<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n<pre class=\"wp-block-code\"><code>&lt;meta name=\"robots\" content=\"index, nofollow\"\/&gt;<\/code><\/pre>\n\n<p>In this case the opposite is true, we tell you that you can index this URL but do NOT follow the links, therefore they will not transmit their value in the usual way.<\/p>\n\n<p>It is the ideal combination when you do not back up the links of a specific URL, imagine pages created by users, for example in a forum.<\/p>\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"noindex_nofollow\"><\/span>noindex, nofollow<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n<pre class=\"wp-block-code\"><code>&lt;meta name=\"robots\" content=\"noindex, nofollow\"\/&gt;<\/code><\/pre>\n\n<p>We avoid indexing and tracking links. It is a form of total blocking of that URL. Its use is not very common. <\/p>\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Index_follow\"><\/span>Index, follow<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n<p>There is a fourth tag which is index, follow but this tag is not necessary to put it because it is the normal behavior, in which a URL is identified, the links are followed and the content is indexed in search engines.<\/p>\n\n<p>However, I think it is positive to mark it explicitly, something that some technologies do not do by default, such as Shopify.  <\/p>\n\n<div style=\"height:30px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Is_there_a_difference_between_robotstxt_and_meta_robots_at_the_crawling_level\"><\/span>Is there a difference between robots.txt and meta robots at the crawling level?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n<p>Of course, remember that <em>robots.txt<\/em> is usually one of the first files that search engines will check.<\/p>\n\n<p>If you mark a <strong>disallow<\/strong> to a directory within that file, in principle Google will not waste time crawling that directory, whereas if it arrives at a URL with the <em>noindex<\/em> tag, it does crawl.<\/p>\n\n<p>In addition, with the robots.txt we can define patterns (imagine blocking directories or subsets of information) while the robots meta tag goes in each URL.<\/p>\n\n<p>What should we take into account about these two ways of controlling crawling and indexing?<\/p>\n\n<p>For example, if we block a directory in the robots.txt, Google will not waste time crawling that section. But if we receive an external link, it is quite likely that it will crawl and even index that section, thus ignoring the instruction we give it.<\/p>\n\n<p>It is therefore important to <strong>leave in the meta robots <\/strong>the directives that we want, in a way to control the final indexing that Google makes of our website.<\/p>\n\n<div style=\"height:30px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Other_directives_for_meta_robots\"><\/span>Other directives for meta robots<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n<p>We can use more elements, some examples:<\/p>\n\n<ul class=\"wp-block-list\">\n<li><strong>archive \/ noarchive<\/strong>: whether or not to store the web content in the internal cache.<\/li>\n\n\n\n<li><strong>noimageindex<\/strong>: not to index the images of the page,<\/li>\n<\/ul>\n\n<p>And some other examples, but with less frequent uses that Google makes available to us in its <a href=\"https:\/\/developers.google.com\/search\/reference\/robots_meta_tag?hl=es-419\" rel=\"noopener\">help page for developers<\/a>.<\/p>\n\n<p>How else can we control indexing?<\/p>\n\n<div style=\"height:30px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"572\" src=\"https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/Meta-etiqueta-cannonicals-1024x572.png\" alt=\"Canonical tag in meta robots  \" class=\"wp-image-10573\" srcset=\"https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/Meta-etiqueta-cannonicals-1024x572.png 1024w, https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/Meta-etiqueta-cannonicals-300x168.png 300w, https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/Meta-etiqueta-cannonicals-768x429.png 768w, https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/Meta-etiqueta-cannonicals-1536x859.png 1536w, https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/Meta-etiqueta-cannonicals-1200x671.png 1200w, https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/Meta-etiqueta-cannonicals.png 1585w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Canonical_label\"><\/span>Canonical label<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n<p>As you can imagine, it has nothing to do with the Catholic Church, canonizations or saints of any kind.<\/p>\n\n<p>In <strong>2009<\/strong>, Google, Yahoo and Microsoft agreed and designed a tag that would allow them to simplify the problem of duplicate content.<\/p>\n\n<p>From an SEO point of view canonical tags are similar to 301. I explain:<\/p>\n\n<p>Imagine that we have a URL:<\/p>\n\n<pre class=\"wp-block-code\"><code>\u00abhttps:\/\/meloquitandelasmanos.com\/zapatochulo\/\u00bb<\/code><\/pre>\n\n<p>And this is an <strong>online store<\/strong> that creates a <strong>variable in the URL<\/strong> if you change an attribute of the product, for example the color, then the URL would be:<\/p>\n\n<pre class=\"wp-block-code\"><code>\u00abhttps:\/\/meloquitandelasmanos.com\/zapatochulo#rojo\/\u00bb<\/code><\/pre>\n\n<p>Or even, making a mailing or creating an <strong>advertising campaign on Facebook<\/strong> for example, leaving the URL with the typical tags that allow you to analyze the campaigns, for example:<\/p>\n\n<pre class=\"wp-block-code\"><code>\u00abhttps:\/\/meloquitandelasmanos.com\/zapatochulo?utm-source=facebook-campaigns=zapatos-verano\/\u00bb<\/code><\/pre>\n\n<p>For practical purposes search engines may interpret these URLs as <strong>duplicate content<\/strong> because it treats them as different URLs with the same content.<\/p>\n\n<p>To do this, we only need to put a tag in the header that indicates the original rel=&#8221;canonical&#8221; and the other variations, the search engines will understand that it is the same page.<\/p>\n\n<p>Note, as you know and I have already mentioned, that sometimes search engines can be &#8220;capricious&#8221; so the canonical tags are taken only as a <strong>suggestion<\/strong>, <strong>not as a directive<\/strong>.<\/p>\n\n<p>If we are in WordPress and you have an SEO plugin, you can rest assured that by default SEO plugins usually add the canonical tag automatically.<\/p>\n\n<p>To check it, just right-click and view the source code of the page. Look for the following tag in the header of your website:<\/p>\n\n<pre class=\"wp-block-code\"><code>&lt;link rel=\"canonical\" href=\"https:\/\/wajari.com\/\" \/><\/code><\/pre>\n\n<p>You can also download some of these browser extensions. Just by clicking on it, you will see all this information:<\/p>\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/chrome.google.com\/webstore\/detail\/seo-meta-in-1-click\/bjogjfinolnhfhkbipphpdlldadpnmhc\" rel=\"noopener\">SEO Meta One<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/chrome.google.com\/webstore\/detail\/seo-pro-extension\/lajfdngpocmchpihjecojjllfjeehgnl\" target=\"_blank\" rel=\"noreferrer noopener\">SEO Pro Extension<\/a><\/li>\n<\/ul>\n\n<p>Curious? Interesting? I hope this was the case.<\/p>\n\n<p>This is a short but basic chapter that covers some basic principles of crawling and indexing that have a lot of relevance in the SEO of our web pages.  <\/p>\n\n<p>So you know what it means when we configure these options in any of the SEO plugins. Live long and prosper!<\/p>\n\n<div style=\"height:50px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n","protected":false},"excerpt":{"rendered":"<p>What are meta robots in SEO? First episode of a series of posts and videos. Basic manuals to go deeper into the world of SEO.  <\/p>\n","protected":false},"author":1,"featured_media":10576,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"_uag_custom_page_level_css":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[87,70],"tags":[71,88],"class_list":["post-10582","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-guides","category-seo-en","tag-google-en","tag-search-engines"],"featured_image_urls_v2":{"full":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots.png",640,426,false],"thumbnail":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots-150x150.png",150,150,true],"medium":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots-300x200.png",300,200,true],"medium_large":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots.png",640,426,false],"large":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots.png",640,426,false],"1536x1536":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots.png",640,426,false],"2048x2048":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots.png",640,426,false]},"post_excerpt_stackable_v2":"<p>What are meta robots in SEO? First episode of a series of posts and videos. Basic manuals to go deeper into the world of SEO.  <\/p>\n","category_list_v2":"<a href=\"https:\/\/wajari.com\/en\/categoria\/guides\/\" rel=\"category tag\">Guides<\/a>, <a href=\"https:\/\/wajari.com\/en\/categoria\/seo-en\/\" rel=\"category tag\">SEO<\/a>","author_info_v2":{"name":"Wajari Vel\u00e1squez","url":"https:\/\/wajari.com\/en\/author\/wajari\/"},"comments_num_v2":"0 comments","jetpack_featured_media_url":"https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots.png","uagb_featured_image_src":{"full":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots.png",640,426,false],"thumbnail":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots-150x150.png",150,150,true],"medium":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots-300x200.png",300,200,true],"medium_large":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots.png",640,426,false],"large":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots.png",640,426,false],"1536x1536":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots.png",640,426,false],"2048x2048":["https:\/\/wajari.com\/wp-content\/uploads\/2023\/08\/2023-08-Meta-robots.png",640,426,false]},"uagb_author_info":{"display_name":"Wajari Vel\u00e1squez","author_link":"https:\/\/wajari.com\/en\/author\/wajari\/"},"uagb_comment_info":0,"uagb_excerpt":"What are meta robots in SEO? First episode of a series of posts and videos. Basic manuals to go deeper into the world of SEO.","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/wajari.com\/en\/wp-json\/wp\/v2\/posts\/10582","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wajari.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wajari.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wajari.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wajari.com\/en\/wp-json\/wp\/v2\/comments?post=10582"}],"version-history":[{"count":2,"href":"https:\/\/wajari.com\/en\/wp-json\/wp\/v2\/posts\/10582\/revisions"}],"predecessor-version":[{"id":10584,"href":"https:\/\/wajari.com\/en\/wp-json\/wp\/v2\/posts\/10582\/revisions\/10584"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/wajari.com\/en\/wp-json\/wp\/v2\/media\/10576"}],"wp:attachment":[{"href":"https:\/\/wajari.com\/en\/wp-json\/wp\/v2\/media?parent=10582"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wajari.com\/en\/wp-json\/wp\/v2\/categories?post=10582"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wajari.com\/en\/wp-json\/wp\/v2\/tags?post=10582"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}