robots-txt-guide

How to use Meta Robots Tags and Robots. txt ?

Posted on Posted in Mobile Marketing and SEO, SEO Tips

What is robots.txt?

Robots.txt is a textual content document webmasters create to instruct net robots (usually seek engine robots) the way to move slowly pages on their website. Robots.txt document is a part of the robots exclusion protocol (rep), a group of web standards that alter how robots move slowly the net, access and index content, and serve that content material up to users. the rep also consists of directives like meta robots, as well as page-, subdirectory-, or web site-huge instructions for how search engines like google should treat hyperlinks (consisting of “comply with” or “nofollow”).
In practice, robots.txt documents imply whether certain user sellers (web-crawling software) can or can't move slowly components of an internet site. these crawl instructions are particular through “disallowing” or “allowing” the conduct of sure (or all) consumer sellers.

Spider Set backs

Search engines like google and yahoo permit you to set web spider priorities. google does not assist the web spider set back command without hold up, however you can lower your web spider precedence interior google webmaster important. google has the best extent of search market proportion in maximum markets, and has one of the maximum green spidering priorities, so that you need to no longer want to exchange google spider precedence.

Browsers that support which robots meta tag values?

The following table shows which search engines aid which values:

Robots Google Yahoo MSN/Live Ask
index Yes Yes Yes Yes
noindex Yes Yes Yes Yes
none Yes Doubt Doubt Yes
follow Yes Doubt Doubt Yes
nofollow Yes Yes Yes Yes
noarchive Yes Yes Yes Yes
nosnippet Yes No No NO
noodip Yes Yes Yes No
noydir Yes Unavailable Unavailable Unavailable
         

Individual robots meta tag values

robots txt tags in SEO

 

 

 

 

 

 

 

 

 

 

 

A proof of all of the one of a kind values you could use inside the robots meta tags:

 1) index
Permit engines like google robots to index the page, you don’t need to include this on your pages, because it’s automatically selected.

2) noindex
Refuse browsers from displaying this web page in their outcomes.

3) noimageindex
Refuse browsers like google and yahoo from crawler snapshots on that page. certainly, if pictures are connected to without setbacks from someplace else, google can still guide them, so the use of an X-Robots-Tag HTTP header is a greater concept.

4) none
An alternative for noindex,nofollow, or essentially announcing to browsers like google: don’t do nothing with this page whatsoever.

5) follow
Notifies the browser crawlers to support the hyperlinks on the page, whether or not it is able to index it or not.

6) nofollow
Notifies the browsers like google crawlers to “notfollow” any hyperlinks at the page in any respect.

7) noarchive
Avertsthe browsers like google from displaying a cached replica of this web page.

8) nocache
identical as noarchive, but only used by “MSN/Live” nosnippet Averts the browsers like google from displaying a preview of this page within the explorer outcomes and averts them from caching the web page.

9) noodp
Used to block browsers from the use of the outline for this page in DMOZ (aka ODP) as the preview for your web page in the explorer consequences. but, DMOZ isn’t there nowadays.

10) noydir
Obstructs yahoo! from the use of the specification for this web page within the yahoo! Directory because the preview in your web page in the explorer consequences. no other browsers like google and yahoo use the yahoo! directory for this motive, so they don’t help the tab. In view that yahoo! closed its listing this tab is deplored, however you would possibly stumble upon it once in a way.

Crawlers can be blocked without delay by the values above, or simply block one crawler by means of being specific.

 

checking when you have a robots.txt record

Unsure if you have a robots.txt record? without a doubt type on your root area, then add /robots.txt to tail end of the URL. As an instance, Moz’s robots record is placed at moz.com/robots.txt.

if no .txt page appears, you do not currently have a (live) robots.txt page.

robots.txt vs meta robots vs x-robots

There are various robots! So we look at the difference between those three kinds of robot instructions firstly, robots.txt is a real text file, whereas meta and x-robots are meta commands. beyond what they virtually are, the 3 all serve distinct functions. robots.txt dominates website online or directory-extensive crawl behaviour, while meta and x-robots can dominate indexation conduct on any particular page (or web page detail) level.

Leave a Reply

Your email address will not be published. Required fields are marked *