The Web Robots FAQ

Original of this document is here ˇ http://info.webcrawler.com/mak/projects/robots/faq.html

</pre>

<BASE HREF="http://info.webcrawler.com/mak/projects/robots/faq.html">

These frequently asked questions about Web robots.
<br>
Send suggestions and comments to
<a href="/mak/mak.html">Martijn Koster</a>.

<p>

<h2><A NAME="toc">Table of Contents</A></h2>

<ol>
<li><a name="about">About WWW robots</a>
<ul>
<li><A HREF="#what">What is a WWW robot?</A>
<li><A HREF="#agent">What is an agent?</A>
<li><A HREF="#engine">What is a search engine?</A>
<li><A HREF="#kinds">What kinds of robots are there?</A>
<li><A HREF="#othernames">So what are Robots, Spiders, Web Crawlers, Worms,
Ants?</a>
<li><A HREF="#bad">Aren't robots bad for the web?</a>

<li><a HREF="#books">Are there any robot books?</a>
<li><A HREF="#info">Where do I find out more about robots?</a>
</ul>

<li><A NAME="indexing">Indexing robots</A>
<ul>
<li><A HREF="#visit">How does a robot decide where to visit?</A>
<li><A HREF="#index">How does an indexing robot decide what to index?</A>
<li><A HREF="#register">How do I register my page with a robot?</A>
</ul>
<li><A NAME="admin">For Server Administrators</A>
<ul>
<li><A HREF="#metoo">How do I know if I've been visited by a robot?</A>
<li><A HREF="#nowwhat">I've been visited by a robot! Now what?</A>
<li><A HREF="#rapidfire">A robot is traversing my whole site too fast!</A>
<li><A HREF="#away">How do I keep a robot off my server?</A>
</ul>
<li><a name="exclusion">Robots exclusion standard</a>
<ul>
<li><A HREF="#log">Why do I find entries for /robots.txt in my log files?</A>
<li><A HREF="#prevent">How do I prevent robots scanning my site?</A>
<li><A HREF="#robotstxt">Where do I find out how /robots.txt files work?</A>
<li><A HREF="#extension">Will the /robots.txt standard be extended?</A>
<li><A HREF="#noindex">What if I cannot make a /robots.txt?</A>
</ul>
<li><a NAME="availability">Availability</a>
<ul>
<li><A HREF="#use">Where can I use a robot?</A>
<li><A HREF="#get">Where can I get a robot?</A>
<li><A HREF="#source">Where can I get the source code for a robot?</A>
<li><A HREF="#writing">I'm writing a robot, what do I need to be careful of?</A>
<li><A HREF="#adding">I've written a robot, how do I list it?</A>
</ul>
</ol>

<hr>

<h2>About Web Robots</h2>

<hr>

<h3><a name="what">What is a WWW robot?</a></h3>

A robot is a program that automatically traverses the Web's hypertext
structure by retrieving a document, and recursively retrieving all
documents that are referenced.

<p>

Note that "recursive" here doesn't limit the definition to any specific
traversal algorithm; even if a robot applies some heuristic to the
selection and order of documents to visit and spaces out requests
over a long space of time, it is still a robot.

<p>

Normal Web browsers are not robots, because the are operated by a human,
and don't automatically retrieve referenced documents (other than
inline images).

<p>

Web robots are sometimes referred to as Web Wanderers, Web Crawlers,
or Spiders. These names are a bit misleading as they give the impression
the software itself moves between sites like a virus; this not the case,
a robot simply visits sites by requesting documents from them.

<hr>

<h3><a name="agent">What is an agent?</a></h3>

The word "agent" is used for lots of meanings in computing these days.
Specifically:

<dl compact>
<dt>
Autonomous agents
<dd>
are programs that do travel between sites, deciding
  themselves when to move and what to do (e.g.
  <a href="http://www.genmagic.com/Telescript/">General Magic's Telescript</a>).
  These can only travel between special servers and are currently not
  widespread in the Internet.
<dt>
Intelligent agents
<dd>
are programs that help users with things, such as
  choosing a product, or guiding a user through form filling, or even
  helping users find things. These have generally little to do with
  networking.
<dt>
User-agent
<dd>
is a technical name for programs that perform networking
  tasks for a user, such as Web User-agents like Netscape Explorer,
  Email User-agent like Qualcomm Eudora etc.
</dl>

<hr>

<h3><a name="engine">What is a search engine?</a></h3>

A search engine is a program that searches through some dataset. In the
context of the Web, the word "search engine" is most often used for search
forms that search through databases of HTML documents gathered by a robot.

<hr>

<h3><a name="kinds">What other kinds of robots are there?</a></h3>

Robots can be used for a number of purposes:

<ul>
<li><a href="#indexing">Indexing</a>
<li>HTML validation
<li>Link validation
<li>"What's New" monitoring
<li>Mirroring
</ul>

See the
<a href="active.html">
list of active robots
</a>
to see what robot does what.
Don't ask me -- all I know is what's on the list...

<hr>

<h3><a name="othernames">So what are Robots, Spiders, Web Crawlers, Worms,
Ants</a></h3>

They're all names for the same sort of thing, with slightly different
connotations:
<dl compact>
<dt>Robots
<dd>the generic name, see <a href="#what">above</a>.
<dt>Spiders
<dd>same as robots, but sounds cooler in the press.
<dt>Worms
<dd>same as robots, although technically a worm is a replicating program,
unlike a robot.
<dt>Web crawlers
<dd>same as robots, but note <a href="http://webcrawler.com/">WebCrawler</a>
is a specific robot
<dt>WebAnts
<dd>distributed cooperating robots.
</dl>

<hr>

<h3><a name="bad">Aren't robots bad for the web?</a></h3>

There are a few reasons people believe robots are bad for the Web:

<ul>
<li>
Certain robot implementations can (and have in the past) overloaded
  networks and servers. This happens especially with people who are
  just starting to write a robot; these days there is sufficient
  information on robots to prevent some of these mistakes.

<li>Robots are operated by humans, who make mistakes in configuration,
  or simply don't consider the implications of their actions.
  This means people need to be careful, and robot authors need to make
  it difficult for people to make mistakes with bad effects

<li>Web-wide indexing robots build a central database of documents,
  which doesn't scale too well to millions of documents on millions
  of sites.
</ul>

But at the same time the majority of robots are well designed,
professionally operated, cause no problems, and provide a valuable service
in the absence of widely deployed better solutions.

<p>

So no, robots aren't inherently bad, nor inherently brilliant,
and need careful attention.

<hr>

<h3><a name="books">Are there any robot books?</a></h3>

Yes:
<dl>
<dt>
<a href="http://www.mcp.com/cgi-bin/bag?isbn=1-56205-463-5&last=/bookstore">Internet Agents: Spiders, Wanderers, Brokers, and Bots</a>
by Fah-Chun Cheong.
<dd>This books covers Web robots, commerce transaction agents,
Mud agents, and a few others. It includes source code for a
simple Web robot based on top of libwww-perl4.
<p>
Its coverage of HTTP, HTML, and Web libraries is a bit too thin to be
a "how to write a web robot" book, but it provides useful background
reading and a good overview of the state-of-the-art, especially if you
haven't got the time to find all the info yourself on the Web.
<p>
Published by <A href="http://www.mcp.com/newriders">New Riders</a>,
ISBN 1-56205-463-5.
<p>

<dt>
<a href="http://www.mcp.com/cgi-bin/bag?isbn=1-57521-016-9&last=/bookstore">
Bots and Other Internet Beasties
</a> by Joseph Williams
<dd>
I haven't seen this myself, but someone said:
<quote>
The William's book 'Bots and other Internet Beasties' was quit
disappointing.  It claims to be a 'how to' book on writing robots,
but my impression is that it is nothing more than a collection
of chapters, written by various people involved in this area and
subsequently bound together.
</quote>

<p>
Published by Sam's, ISBN: 1-57521-016-9
<p>
</dd>

<dt>
<a href="http://www.ora.com/catalog/webclient/">Web
Client Programming with Perl</a> by Clinton Wong
<dd>
This O'Reilly book is planned for Fall 1996, check the
<a href="http://www.ora.com">O'Reilly Web Site</a> for
the current status. It promises to be a practical book,
but I haven't seen it yet.
</dd>
</dl>

A few others can be found on the
<a href="http://www.ee.mcgill.ca/~belmarc/agent_faq.html">
The Software Agents Mailing List FAQ
</a>
<hr>

<h3><a name="info">Where do I find out more about robots?</a></h3>

There is a Web robots home page on:

<a href="http://info.webcrawler.com/mak/projects/robots/robots.html">
http://info.webcrawler.com/mak/projects/robots/robots.html
</a>

<p>

While this is hosted at one of the major robots' site, it is
an unbiased and reasoneably comprehensive collection of information
which is maintained by Martijn Koster &lt;m.koster@webcrawler.com&gt;.

<p>

Of course the latest version of this FAQ is there.

<p>

You'll also find details and an archive of the robots mailing
list, which is intended for technical discussions about robots.

<hr>

<h2>Indexing robots</h2>

<hr>

<h3><a name="visit">How does a robot decide where to visit?</a></h3>

This depends on the robot, each one uses different strategies.
In general they start from a historical list of URLs, especially
of documents with many links elsewhere, such as server lists,
"What's New" pages, and the most popular sites on the Web.

<p>

Most indexing services also allow you to submit URLs manually,
which will then be queued and visited by the robot.

<p>

Sometimes other sources for URLs are used, such as scanners
through USENET postings, published mailing list achives etc.

<p>

Given those starting points a robot can select URLs to visit
and index, and to parse and use as a source for new URLs.

<h3><a name="index">How does an indexing robot decide what to index?</a></h3>

If an indexing robot knows about a document, it may decide to
parse it, and insert it into its database. How this is done
depends on the robot: Some robots index the HTML
Titles, or the first few paragraphs, or parse the entire
HTML and index all words, with weightings depending on HTML
constructs, etc. Some parse the META tag, or other special
hidden tags.

<p>

We hope that as the Web evolves more facilities becomes available
to efficiently associate meta data such as indexing information
with a document. This is being worked on...

<hr>

<h3><a name="register">How do I register my page with a robot?</a></h3>

You guessed it, it depends on the service :-) Most services have
a link to a URL submission form on their search page.

<p>

Fortunately you don't have to submit your URL to every service
by hand: <a href="http://www.submit-it.com/">Submit-it</a>
&lt;URL: http://www.submit-it.com/&gt; will do it for you.

<hr>

<h2>For Server Administrators</h2>

<hr>

<h3><a name="metoo">How do I know if I've been visited by a robot?</a></h3>

You can check your server logs for sites that retrieve many
documents, especially in a short time.

<p>

If your server supports User-agent logging you can check for
retrievals with unusual User-agent heder values.

<p>

Finally, if you notice a site repeatedly checking for the file
'/robots.txt' chances are that is a robot too.

<hr>

<h3><a name="nowwhat">I've been visited by a robot! Now what?</a></h3>

Well, nothing :-) The whole idea is they are automatic; you don't
need to do anything.

<p>

If you think you have discovered a new robot (ie one that is not
listed on <a href="active.html">
the list of active robots</a>, and it does more than sporadic visits,
drop me a line so I can make a note of it for future reference.
But please don't tell me about every robot that happens to drop by!

<hr>

<h3><a name="rapidfire">A robot is traversing my whole site too fast!</a></h3>

This is called "rapid-fire", and people usually notice it if they're
monitoring or analysing an access log file.

<p>

First of all check if it is a problem by checking the load of your server,
and monitoring your servers' error log, and concurrent connections if
you can. If you have a medium or high performance server, it is quite
likely to be able to cope a high load of even several requests per second,
especially if the visits are quick.

<p>

However you may have problems if you have a low performance site, such as
your own desktop PC or Mac you're working on, or you run low performance
server software, or if you have many long retrievals (such as CGI scripts
or large documents). These problems manifest themselves in refused
connections, a high load, performance slowdowns, or in extreme cases a
system crash.

<p>

If this happens, there are a few things you should do. Most importantly,
start logging information: when did you notice, what happened, what do
your logs say, what are you doing in response etc; this helps investigating
the problem later. Secondly, try and find out where the robot came from,
what IP addresses or DNS domains, and see if they are mentioned in the
<a href="active.html">
list of active robots</a>. If you can identify a site this way, you can
email the person responsible, and ask them what's up. If this doesn't help,
try their own site for telephone numbers, or mail postmaster at their
domain.

<p>

If the robot is not on the list, mail me with all the information you
have collected, including actions on your part. If I can't help, at least
I can make a note of it for others.

<hr>

<h3><a name="away">How do I keep a robot off my server?</a></h3>

Read the next section...

<hr>

<h2>Robots exclusion standard</h2>

<hr>

<h3><a name="log">Why do I find entries for /robots.txt in my log
files?</a></h3>

They are probably from robots trying to see if you have specified
any rules for them using the <a href="norobots.html">Standard
for Robot Exclusion</a>, see also <a href="#robots.txt">below</a>.

<p>

If you don't care about robots and want to prevent the messages
in your error logs, simply create an empty file called robots.txt
in the root level of your server.

<p>

Don't put any HTML or English language "Who the hell are you?"
text in it -- it will probably never get read by anyone :-)

<hr>

<h3><a name="prevent">How do I prevent robots scanning my site?</a></h3>

The quick way to prevent robots visiting your site is put these
two lines into the /robots.txt file on your server:

<blockquote>
<pre>
User-agent: *
Disallow: /
</pre>
</blockquote>

but its easy to <a href="#robotstxt">be more selective</a> than that.

<hr>

<h3><a name="robotstxt">Where do I find out how /robots.txt files work?</a></h3>

You can read <a href="norobots.html">the whole standard specification</a>
but the basic concept is simple: by writing a structured text
file you can indicate to robots that certain parts of your
server are off-limits to some or all robots. It is best explained
with an example:
<blockquote>
<pre>
# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism

User-agent: webcrawler
Disallow:

User-agent: lycra
Disallow: /

User-agent: *
Disallow: /tmp
Disallow: /logs
</pre>
</blockquote>
The first two lines, starting with '#', specify a comment

<p>

The first paragraph specifies that the robot called 'webcrawler'
has nothing disallowed: it may go anywhere.

<p>

The second paragraph indicates that the robot called 'lycra'
has all relative URLs starting with '/' disallowed.
Because all relative URL's on a server start with '/',
this means the entire site is closed off.

<p>

The third paragraph indicates that all other robots should not
visit URLs starting with /tmp or /log. Note the '*' is a special
token; its not a regular expression.

<p>

Two common errors:
<ul>
<li>
Regular expressions are _not_ supported: instead of
'Disallow: /tmp/*' just say 'Disallow: /tmp'.
<li>
You shouldn't put more than one path on a Disallow line (this may
change in a future version of the spec)
</ul>

<hr>

<h3><a name="extension">Will the /robots.txt standard be extended?</a></h3>

Probably... there are some ideas floating around. They haven't
made it into a coherent proposal because of time constraints,
and because there is little pressure. Mail suggestions to the
robots mailing list, and check the robots home page for work
in progress.

<hr>

<h3><a name="noindex">What if I can't make a /robots.txt file?</a></h3>

Sometimes you cannot make a /robots.txt file, because you don't
administer the entire server. All is not lost: there is a new
<a href="http://www.w3.org/pub/WWW/Search/9605-Indexing-Workshop/ReportOutcomes/Spidering.txt">standard</a>
for using HTML META tags to keep robots out of your documents.

<p>

The basic idea is that if you include a tag like:
<blockquote>
<pre>
&lt;META NAME="ROBOTS" CONTENT="NOINDEX"&gt;
</pre>
</blockquote>
in your HTML document, that document won't be indexed.
<p>
If you do:
<blockquote>
<pre>
&lt;META NAME="ROBOTS" CONTENT="NOFOLLOW"&gt;
</pre>
</blockquote>
the links in that document will not be parsed by the robot.
<hr>

<h2>Availability</h2>

<hr>

<h3><a name="use">Where can I use a robot?</a></h3>

If you mean a search service, check out the various directory pages
on the Web, such as
Netscape's
<a href="http://home.netscape.com/home/internet-search.html">
Exploring the Net</a>
or try one of the Meta search services such as
<a href="http://metasearch.com/">MetaSearch</a>
<hr>

<h3><a name="get">Where can I get a robot?</a></h3>

Well, you can have a look at the list of robots; I'm starting to
indicate their public availability slowly.

<p>

In the meantime, two indexing robots that you should be able to
get hold of are Harvest (free), and Verity's.
<hr>

<h3><a name="source">Where can I get the source code for a robot?</a></h3>

See <A href="#get">above</a> -- some may be willing to give out source code.
<p>
Alternatively check out the libwww-perl5 package, that has a simple example.
<hr>

<h3><a name="writing">I'm writing a robot, what do I need to be careful
of?</a></h3>

Lots. First read through all the stuff on
<a href="http://info.webcrawler.com/mak/projects/robots/robots.html">
the robot page</a>
then read the proceedings of past WWW Conferences, and the
complete HTTP and HTML spec. Yes; it's a lot of work :-)

<hr>

<h3><a name="adding">I've written a robot, how do I list it?</a></h3>

Simply fill in a form you can find on
<a href="http://info.webcrawler.com/mak/projects/robots/active.html">
The Web Robots Database</a> and email it to me.

<hr>
<div align=right>
<address>
<small>
<a href="/mak/mak.html">Martijn Koster</a>
</small>
</address>
</div>

        Neskol'ko slov o tom, kak rabotayut roboty (spiders) poiskovyh mashin
<I>Andrej Alikberov, <a href="http://www.citmgu.ru">Centr Informacionnyh Tehnologij</a></I></CENTER>
<ul type=disk>
<li><a href="#0">Vvedenie</a>
<ul type=disk>
<li><a href="#1">Format fajla /robots.txt</a>
<li><a href="#2">Zapisi (records) fajla /robots.txt</a>
<li><a href="#3">Parametry</a>
<li><a href="#4">Rasshirennye kommentarii formata</a>
<li><a href="#5">Primery</a>
<li><a href="#6">Nekotorye problemy, svyazannye s poiskovymi robotami</a>
<li><a href="#7">Ispol'zovanie meta-tagov HTML</a>
</ul>
<li><a href="#8">ROBOTS meta-tagi</a>
<ul type=disk>
<li><a href="#9">KEYWORDS meta-tag</a>
<li><a href="#10">DESCRIPTION meta-tag</a>
<li><a href="#11">Predpolagaemye varianty isklyucheniya povtornyh poseshchenij s pomoshch'yu
meta-tagov HTML</a>
<li><a href="#12">Indeksirovat' dokument odin raz ili delat' eto regulyarno?</a>
<li><a href="#13">Kak isklyuchit' indeksirovanie generiruemyh stranic ili dublirovanie
dokumentov, esli est' zerkala servera?</a>
<li><a href="#14">Istochniki</a>
</ul></ul>
<center><a name="0"></a><p><b>Vvedenie</b></center>
<P align=justify>
|ta stat'ya vovse ne yavlyaetsya popytkoj ob®yasnit', kak rabotayut
poiskovye mashiny voobshche (eto know-how
ih proizvoditelej). Odnako, po moemu mneniyu, ona pomozhet
ponyat' kak mozhno upravlyat' povedeniem poiskovyh robotov (wanderers,
spiders, robots - programmy, s pomoshch'yu kotoryh ta ili inaya
poiskovaya sistema obsharivaet set' i indeksiruet vstrechayushchiesya
dokumenty) i kak pravil'no
postroit' strukturu servera i soderzhashchihsya na nem dokumentov,
chtoby Vash server legko i horosho indeksirovalsya.
<P align=justify >
Pervoj prichinoj togo, chto ya reshilsya napisat' etu stat'yu, yavilsya
sluchaj, kogda ya issledoval fajl logov dostupa k moemu serveru
i obnaruzhil tam sleduyushchie dve stroki:
</p><p>
lycosidae.lycos.com - - [01/Mar/1997:21:27:32 -0500] &quot;GET /robots.txt HTTP/1.0&quot; 404 -<br>
lycosidae.lycos.com - - [01/Mar/1997:21:27:39 -0500] &quot;GET / HTTP/1.0&quot; 200 3270

<p align=justify >
to est' Lycos obratilsya k
moemu serveru, na pervyj zapros poluchil, chto fajla /robots.txt
net, obnyuhal pervuyu stranicu, i otvalil. Estestvenno, mne eto
ne ponravilos', i ya nachal vyyasnyat' chto k chemu.
<P align=justify >
Okazyvaetsya, vse &quot;umnye&quot; poiskovye mashiny snachala obrashchayutsya
k etomu fajlu, kotoryj dolzhen prisutstvovat' na kazhdom servere.
|tot fajl opisyvaet prava dostupa dlya poiskovyh robotov, prichem
sushchestvuet vozmozhnost' ukazat' dlya razlichnyh robotov raznye prava.
Dlya nego sushchestvuet standart pod nazvaniem <a href="rbtspec.html">Standart
for Robot Exclusion</a>.
<P align=justify >
Po mneniyu Luisa Mon'e (Louis Monier, Altavista),
tol'ko 5% vseh sajtov v nastoyashchee vremya imeet ne pustye fajly
/robots.txt esli voobshche oni (eti fajly) tam sushchestvuyut. |to podtverzhdaetsya
informaciej, sobrannoj pri nedavnem issledovanii logov raboty
robota Lycos. SHarl' Kollar (Charles
P.Kollar, Lycos) pishet, chto tol'ko 6% ot vseh zaprosov
na predmet /robots.txt imeyut kod rezul'tata 200. Vot neskol'ko
prichin, po kotorym eto proishodit:
<UL>
<LI>lyudi, kotorye ustanavlivayut Veb-servera, prosto ne znayut ni
ob etom standarte, ni o neobhodimosti sushchestvovaniya fajla /robots.txt.
<LI>ne obyazatel'no chelovek, installirovavshij Veb-server, zanimaetsya
ego napolneniem, a tot, kto yavlyaetsya vebmasterom, ne imeet dolzhnogo
kontakta s administratorom samoj &quot;zhelezyaki&quot;.
<LI>eto chislo otrazhaet chislo sajtov, kotorye dejstvitel'no nuzhdayutsya
v isklyuchenii lishnih zaprosov robotov, poskol'ku ne na vseh serverah
imeetsya takoj sushchestvennyj trafik, pri kotorom poseshchenie servera
poiskovym robotom, stanovitsya zametnym dlya prostyh pol'zovatelej.
</UL>
<P><a name="1"></a>
<B>Format fajla /robots.txt.</B>
<P align=justify >
Fajl /robots.txt prednaznachen dlya ukazaniya vsem poiskovym robotam
(spiders) indeksirovat' informacionnye servera tak, kak opredeleno
v etom fajle, t.e. tol'ko te direktorii i fajly servera, kotorye
NE opisany v /robots.txt. |to fajl dolzhen soderzhat' 0 ili bolee
zapisej, kotorye svyazany s tem ili inym robotom (chto opredelyaetsya
znacheniem polya agent_id), i ukazyvayut dlya kazhdogo robota ili dlya
vseh srazu chto imenno im NE NADO indeksirovat'. Tot, kto pishet
fajl /robots.txt, dolzhen ukazat' podstroku Product Token polya
User-Agent, kotoruyu kazhdyj robot vydaet na HTTP-zapros indeksiruemogo
servera. Naprimer, nyneshnij robot Lycos na takoj zapros vydaet
v kachestve polya User-Agent:
<Pre>
	Lycos_Spider_(Rex)/1.0 libwww/3.1
</pre>
<P align=justify >
Esli robot Lycos ne nashel svoego opisaniya v /robots.txt - on postupaet
tak, kak schitaet nuzhnym. Kak tol'ko robot Lycos &quot;uvidel&quot;
v fajle /robots.txt opisanie dlya sebya - on postupaet tak, kak
emu predpisano.
<P align=justify >
Pri sozdanii fajla /robots.txt sleduet uchityvat' eshche odin faktor
- razmer fajla. Poskol'ku opisyvaetsya kazhdyj fajl, kotoryj ne
sleduet indeksirovat', da eshche dlya mnogih tipov robotov otdel'no,
pri bol'shom kolichestve ne podlezhashchih indeksirovaniyu fajlov razmer
/robots.txt stanovitsya slishkom bol'shim. V etom sluchae sleduet
primenyat' odin ili neskol'ko sleduyushchih sposobov sokrashcheniya razmera
/robots.txt:
<UL>
<LI>ukazyvat' direktoriyu, kotoruyu ne sleduet indeksirovat', i,
sootvetstvenno, ne podlezhashchie indeksirovaniyu fajly raspolagat'
imenno v nej
<LI>sozdavat' strukturu servera s uchetom uproshcheniya opisaniya isklyuchenij
v /robots.txt
<LI>ukazyvat' odin sposob indeksirovaniya dlya vseh agent_id
<LI>ukazyvat' maski dlya direktorij i fajlov
</UL>
<P><a name="2"></a>
<B>Zapisi (records) fajla /robots.txt</B>
<P>
Obshchee opisanie formata zapisi.
<P>
<TT>[ # comment string NL
]*</TT>
<P>
<TT>User-Agent: [ [ WS ]+
agent_id ]+ [ [ WS ]* # comment string ]? NL</TT>
<P>
<TT>[ # comment string NL
]*</TT>
<P>
<TT>Disallow: [ [ WS ]+ path_root
]* [ [ WS ]* # comment string ]? NL</TT>
<P>
<TT>[</TT>
<P>
<TT>  # comment string NL</TT>
<P>
<TT>|</TT>
<P>
<TT>  Disallow: [ [ WS ]+
path_root ]* [ [ WS ]* # comment string ]? NL</TT>
<P>
<TT>]*</TT>
<P>
<TT>[ NL ]+</TT>
<P><a name="3"></a>
<B>Parametry</B>
<P>
Opisanie parametrov, primenyaemyh v zapisyah /robots.txt
<P align=justify >
[...]+ Kvadratnye skobki so sleduyushchim za nimi znakom + oznachayut,
chto v kachestve parametrov dolzhny byt' ukazany odin ili neskol'ko
terminov.
<P align=justify >
Naprimer, posle &quot;User-Agent:&quot; cherez probel mogut byt'
ukazany odin ili neskol'ko agent_id.
<P align=justify >
[...]* Kvadratnye skobki so sleduyushchim za nimi znakom * oznachayut,
chto v kachestve parametrov mogut byt' ukazany nol' ili neskol'ko
terminov.
<P align=justify >
Naprimer, Vy mozhete pisat' ili ne pisat' kommentarii.
<P align=justify >
[...]? Kvadratnye skobki so sleduyushchim za nimi znakom ? oznachayut,
chto v kachestve parametrov mogut byt' ukazany nol' ili odin termin.
<P align=justify >
Naprimer, posle &quot;User-Agent: agent_id&quot; mozhet byt' napisan
kommentarij.
<P align=justify >
..|.. oznachaet ili to, chto do cherty, ili to, chto posle.
<P align=justify >
WS odin iz simvolov - probel (011) ili tabulyaciya (040)
<P align=justify >
NL odin iz simvolov - konec stroki (015) , vozvrat karetki (012)
ili oba etih simvola (Enter)
<P align=justify >
User-Agent: klyuchevoe slovo (zaglavnye i propisnye bukvy roli ne
igrayut).
<P align=justify >
Parametrami yavlyayutsya agent_id poiskovyh robotov.
<P align=justify >
Disallow: klyuchevoe slovo (zaglavnye i propisnye bukvy roli ne
igrayut).
<P align=justify >
Parametrami yavlyayutsya polnye puti k neindeksiruemym fajlam ili
direktoriyam
<P align=justify >
# nachalo stroki kommentariev, comment string - sobstvenno telo
kommentariya.
<P align=justify >
agent_id lyuboe kolichestvo simvolov, ne vklyuchayushchih WS i NL, kotorye
opredelyayut agent_id razlichnyh poiskovyh robotov. Znak * opredelyaet
vseh robotov srazu.
<P align=justify >
path_root lyuboe kolichestvo simvolov, ne vklyuchayushchih WS i NL, kotorye
opredelyayut fajly i direktorii, ne podlezhashchie indeksirovaniyu.
<P><a name="4"></a>
<B>Rasshirennye kommentarii formata.</B>
<P align=justify >
Kazhdaya zapis' nachinaetsya so stroki User-Agent, v kotoroj opisyvaetsya
kakim ili kakomu poiskovomu robotu eta zapis' prednaznachaetsya.
Sleduyushchaya stroka: Disallow. Zdes' opisyvayutsya ne podlezhashchie indeksacii
puti i fajly. KAZHDAYA zapis' DOLZHNA imet' kak minimum eti dve stroki
(lines). Vse ostal'nye stroki yavlyayutsya opciyami. Zapis' mozhet soderzhat'
lyuboe kolichestvo strok kommentariev. Kazhdaya stroka kommentariya
dolzhna nachinat'sya s simvola # . Stroki kommentariev mogut byt'
pomeshcheny v konec strok User-Agent i Disallow. Simvol # v konce
etih strok inogda dobavlyaetsya dlya togo, chtoby ukazat' poiskovomu
robotu, chto dlinnaya stroka agent_id ili path_root zakonchena. Esli
v stroke User-Agent ukazano neskol'ko agent_id, to uslovie path_root
v stroke Disallow budet vypolneno dlya vseh odinakovo. Ogranichenij
na dlinu strok User-Agent i Disallow net. Esli poiskovyj robot
ne obnaruzhil v fajle /robots.txt svoego agent_id, to on ignoriruet
/robots.txt.
<P align=justify >
Esli ne uchityvat' specifiku raboty kazhdogo poiskovogo robota,
mozhno ukazat' isklyucheniya dlya vseh robotov srazu. |to dostigaetsya
zadaniem stroki
<Pre>
	User-Agent: *
</pre>
<P align=justify >
Esli poiskovyj robot obnaruzhit v fajle /robots.txt neskol'ko zapisej
s udovletvoryayushchim ego znacheniem agent_id, to robot volen vybirat'
lyubuyu iz nih.
<P align=justify >
Kazhdyj poiskovyj robot budet opredelyat' absolyutnyj URL dlya chteniya
s servera s ispol'zovaniem zapisej /robots.txt. Zaglavnye i strochnye
simvoly v path_root IMEYUT znachenie.
<P><a name="5"></a>
<B>Primery.</B>
<P>
<I>Primer 1:</I>
<P>
<TT>User-Agent: *</TT>
<P>
<TT>Disallow: /<BR>
</TT>
<P>
<TT>User-Agent: Lycos</TT>
<P>
<TT>Disallow: /cgi-bin/ /tmp/
<BR>
</TT>
<P>
V primere 1 fajl /robots.txt soderzhit dve zapisi. Pervaya otnositsya
ko vsem poiskovym robotam i zapreshchaet indeksirovat' vse fajly.
Vtoraya otnositsya k poiskovomu robotu Lycos i pri indeksirovanii
im servera zapreshchaet direktorii /cgi-bin/ i /tmp/, a ostal'nye
- razreshaet. Takim obrazom server budet proindeksirovan tol'ko
sistemoj Lycos.
<P>
<I>Primer 2:</I>
<P>
<TT>User-Agent: Copernicus
Fred</TT>
<P>
<TT>Disallow:<BR>
</TT>
<P>
<TT>User-Agent: * Rex</TT>
<P>
<TT>Disallow: /t<BR>
</TT>
<P>
V primere 2 fajl /robots.txt soderzhit dve zapisi. Pervaya razreshaet
poiskovym robotam Copernicus i Fred indeksirovat' ves' server.
Vtoraya - zapreshchaet vsem i osebenno robotu Rex indeksirovat' takie
direktorii i fajly, kak /tmp/, /tea-time/, /top-cat.txt, /traverse.this
i t.d. |to kak raz sluchaj zadaniya maski dlya direktorij i fajlov.
<P>
<I>Primer 3:</I>
<P>
<TT># This is for every spider!</TT>
<P>
<TT>User-Agent: *</TT>
<P>
<TT># stay away from this</TT>
<P>
<TT>Disallow: /spiders/not/here/
#and everything in it</TT>
<P>
<TT>Disallow: # a little nothing</TT>
<P>
<TT>Disallow: #This could
be habit forming!</TT>
<P>
<TT># Don't comments make
code much more readable!!!</TT>
<P>
V primere 3 - odna zapis'. Zdes' vsem robotam zapreshchaetsya indeksirovat'
direktoriyu /spiders/not/here/, vklyuchaya takie puti i fajly kak
/spiders/not/here/really/, /spiders/not/here/yes/even/me.html.
Odnako syuda ne vhodyat /spiders/not/ ili /spiders/not/her (v direktorii
'/spiders/not/').
<P><a name="6"></a>
<B>Nekotorye problemy, svyazannye s poiskovymi robotami.</B>
<P>
<I>Nezakonchennost' standarta (</I>Standart
for Robot Exclusion<I>).</I>
<P align=justify >
K sozhaleniyu, poskol'ku poiskovye sistemy poyavilis' ne tak davno,
standart dlya robotov nahoditsya v stadii razrabotki, dorabotki,
nu i t.d. |to oznachaet, chto v budushchem sovsem neobyazatel'no poiskovye
mashiny budut im rukovodstvovat'sya.
<P>
<I>Uvelichenie trafika.</I>
<P align=justify >
|ta problema ne slishkom aktual'na dlya rossijskogo sektora Internet,
poskol'ku ne tak uzh mnogo v Rossii serverov s takim ser'eznym
trafikom, chto poseshchenie ih poiskovym robotom budet meshat' obychnym
pol'zovatelyam. Sobstvenno, fajl /robots.txt
dlya togo i prednaznachen, chtoby ogranichivat' dejstviya robotov.
<P>
<I>Ne vse poiskovye roboty ispol'zuyut /robots.txt.</I>
<P align=justify >
Na segodnyashnij den' etot fajl obyazatel'no zaprashivaetsya poiskovymi
robotami tol'ko takih sistem kak Altavista,
Excite, Infoseek, Lycos, OpenText i WebCrawler.
<P><a name="7"></a>
<B>Ispol'zovanie meta-tagov HTML.</B>
<P align=justify >
Nachal'nyj proekt, kotoryj byl sozdan v rezul'tate soglashenij mezhdu
programmistami nekotorogo chisla kommercheskih indeksiruyushchih organizacij
(Excite, Infoseek, Lycos, Opentext i  WebCrawler) na nedavnem
sobranii Distributing Indexing Workshop (W3C) , nizhe.
<P align=justify >
Na etom sobranii obsuzhdalos' ispol'zovanie meta-tagov HTML dlya
upravleniya povedeniem poiskovyh robotov, no okonchatel'nogo soglasheniya
dostignuto ne bylo. Byli opredeleny sleduyushchie problemy dlya obsuzhdeniya
v budushchem:
<UL>
<LI>Neopredelennosti v specifikacii fajla /robots.txt
<LI>Tochnoe opredelenie ispol'zovaniya meta-tagov HTML, ili dopolnitel'nye
polya v fajle /robots.txt
<LI>Informaciya &quot;Please visit&quot;
<LI>Tekushchij kontrol' informacii: interval ili maksimum otkrytyh
soedinenij s serverom, pri kotoryh mozhno nachinat' indeksirovat'
server.
</UL>
<hr width=90%>
<P><a name="8"></a>
<CENTER><B>ROBOTS meta-tagi</B></CENTER>
<P align=justify >
|tot tag prednaznachen dlya pol'zovatelej, kotorye ne mogut kontrolirovat'
fajl /robots.txt na svoih veb-sajtah. Tag pozvolyaet zadat' povedenie
poiskovogo robota dlya kazhdoj HTML-stranicy, odnako pri etom nel'zya
sovsem izbezhat' obrashcheniya robota k nej (kak vozmozhno ukazat' v
fajle /robots.txt).
<P>
&lt;META NAME=&quot;ROBOTS&quot; CONTENT=&quot;robot_terms&quot;&gt;
<P align=justify >
robot_terms - eto razdelennyj zapyatymi spisok sleduyushchih klyuchevyh
slov (zaglavnye ili strochnye simvoly roli ne igrayut): ALL, NONE,
INDEX, NOINDEX, FOLLOW, NOFOLLOW.<BR>
<P align=justify >
NONE - govorit vsem robotam ignorirovat' etu stranicu pri indeksacii
(ekvivalentno odnovremennomu ispol'zovaniyu klyuchevyh slov NOINDEX,
NOFOLLOW).
<P align=justify >
ALL - razreshaet indeksirovat' etu stranicu i vse ssylki iz nee
(ekvivalentno odnovremennomu ispol'zovaniyu klyuchevyh slov INDEX,
FOLLOW).
<P align=justify >
INDEX - razreshaet indeksirovat' etu stranicu
<P align=justify >
NOINDEX - nerazreshaet indeksirovat' etu stranicu
<P align=justify >
FOLLOW - razreshaet indeksirovat' vse ssylki iz etoj stranicy
<P align=justify >
NOFOLLOW - nerazreshaet indeksirovat' ssylki iz etoj stranicy
<P align=justify >
Esli etot meta-tag propushchen ili ne ukazany robot_terms, to po
umolchaniyu poiskovyj robot postupaet kak esli by byli ukazany robot_terms=
INDEX, FOLLOW (t.e. ALL).
Esli v CONTENT obnaruzheno
klyuchevoe slovo ALL, to robot
postupaet sootvetstvenno, ignoriruya vozmozhno ukazannye drugie
klyuchevye slova.. Esli v CONTENT imeyutsya
protivopolozhnye po smyslu klyuchevye slova, naprimer, FOLLOW,
NOFOLLOW, to robot postupaet po svoemu usmotreniyu (v etom
sluchae FOLLOW).
<P align=justify >
Esli robot_terms soderzhit tol'ko NOINDEX,
to ssylki s etoj stranicy ne indeksiruyutsya. Esli robot_terms soderzhit
tol'ko NOFOLLOW, to stranica
indeksiruetsya, a ssylki, sootvetstvenno, ignoriruyutsya.
<P><a name="9"></a>
<B>KEYWORDS meta-tag.</B>
<P>
&lt;META NAME=&quot;KEYWORDS&quot;
CONTENT=&quot;phrases&quot;&gt;
<P align=justify >
phrases - razdelennyj zapyatymi
spisok slov ili slovosochetanij (zaglavnye i strochnye simvoly roli
ne igrayut), kotorye pomogayut indeksirovat' stranicu (t.e. otrazhayut
soderzhanie stranicy). Grubo govorya, eto te slova, v otvet na kotorye
poiskovaya sistema vydast etot dokument.
<P><a name="10"></a>
<B>DESCRIPTION meta-tag.</B>
<P>
&lt;META NAME=&quot;DESCRIPTION&quot;
CONTENT=&quot;text&quot;&gt;
<P align=justify >
text - tot tekst, kotoryj
budet vyvodit'sya v summarnom otvete na zapros pol'zovatelya k poiskovoj
sisteme. Sej tekst ne dolzhen soderzhat' tagov razmetki i logichnee
vsego vpisat' v nego smysl dannogo dokumenta na paru-trojku strok.
<P><a name="11"></a>
<B>Predpolagaemye varianty isklyucheniya povtornyh poseshchenij s pomoshch'yu
meta-tagov HTML</B>
<P align=justify >
Nekotorye kommercheskie poiskovye roboty uzhe ispol'zuyut meta-tagi,
pozvolyayushchie osushchestvlyat' &quot;svyaz'&quot; mezhdu robotom i vebmasterom.
Altavista ispol'zuet KEYWORDS
meta-tag, a Infoseek ispol'zuet
KEYWORDS i
DESCRIPTION meta-tagi.
<P><a name="12"></a>
<B>Indeksirovat' dokument odin raz ili delat' eto regulyarno?</B>
<P align=justify >
Vebmaster mozhet &quot;skazat'&quot; poiskovomu robotu ili fajlu
bookmark pol'zovatelya, chto
soderzhimoe togo ili inogo fajla budet izmenyat'sya. V etom sluchae
robot ne budet sohranyat' URL,
a brouzer pol'zovatelya vneset ili ne vneset eto fajl v bookmark.
Poka eta informaciya opisyvaetsya tol'ko v fajle /robots.txt,
pol'zovatel' ne budet znat' o tom, chto eta stranica budet izmenyat'sya.
<P align=justify >
Meta-tag DOCUMENT-STATE mozhet
byt' polezen dlya etogo. Po umolchaniyu, etot meta-tag prinimaetsya
s CONTENT=STATIC.
<P>
&lt;META NAME=&quot;DOCUMENT-STATE&quot;
CONTENT=&quot;STATIC&quot;&gt;
<P>
&lt;META NAME=&quot;DOCUMENT-STATE&quot;
CONTENT=&quot;DYNAMIC&quot;&gt;
<P><a name="13"></a>
<B>Kak isklyuchit' indeksirovanie generiruemyh stranic ili dublirovanie
dokumentov, esli est' zerkala servera?</B>
<P align=justify >
Generiruemye stranicy - stranicy, porozhdaemye dejstviem CGI-skriptov.
Ih navernyaka ne sleduet indeksirovat', poskol'ku esli poprobovat'
provalit'sya v nih iz poiskovoj sistemy, budet vydana oshibka. CHto
kasaetsya zerkal, to negozhe, kogda vydayutsya dve raznye ssylki na
raznye servera, no s odnim i tem zhe soderzhimym. CHtoby etogo izbezhat',
sleduet ispol'zovat' meta-tag URL
s ukazaniem absolyutnogo URL
etogo dokumenta (v
sluchae zerkal - na sootvetstvuyushchuyu stranicu glavnogo servera).
<P>
&lt;META NAME=&quot;URL&quot; CONTENT=&quot;absolute_url&quot;&gt;
<P><a name="14"></a>
<B>Istochniki</B>
<ol>
<li>Charles P.Kollar, John R.R. Leavitt,
Michael Mauldin, <B>Robot Exclusion Standard Revisited</B><B>,
</B><a href="http://www.kollar.com/robots.html">www.kollar.com/robots.html</a><P>
<Li>Martijn Koster, <B>Standard for
robot exclusion</B>, <a href="http://info.webcrawler.com/mak/projects/robots/robots.html">info.webcrawler.com/mak/projects/robots/robots.html</a>
</ol>

        Standart isklyuchenij dlya robotov
Standard for robot exclusion

<CENTER><P><I>Martijn Koster , perevod A. Alikberova</I>
</FONT></P></CENTER>

<UL>
<LI><A HREF="#1">Status etogo dokumenta</A></LI>
<LI><A HREF="#2">Vvedenie</A></LI>
<LI><A HREF="#3">Naznachenie</A></LI>
<LI><A HREF="#4">Format</A></LI>
<LI><A HREF="#5">Primery</A></LI>
<LI><A HREF="#6">Primechaniya perevodchika</A></LI>
<LI><A HREF="#7">Adresa avtorov</A></LI>
</UL>

<P><A NAME="1"></A><B>Status etogo dokumenta</B> </P>

<P align=justify>|tot dokument sostavlen 30 iyulya 1994 goda po materialam
obsuzhdenij v telekonferencii robots-request@nexor.co.uk (sejchas konferenciya
perenesena na WebCrawler. Podrobnosti sm. Robots pages at WebCrawler <A HREF="http://info.webcrawler.com/mak/projects/robots/">info.webcrawler.com/mak/projects/robots/</A>)
mezhdu bol'shinstvom proizvoditelej poiskovyh robotov i drugimi zainteresovannymi
lyud'mi.Takzhe eta tema otkryta dlya obsuzhdeniya v telekonferencii Technical
World Wide Web www-talk@info.cern.ch Sej dokument osnovan na predydushchem
rabochem proekte pod takim zhe nazvaniem.</P>

<P align=justify>|tot dokument ne yavlyaetsya oficial'nym ili ch'im-libo korporativnym
standartom, i ne garantiruet togo, chto vse nyneshnie i budushchie poiskovye
roboty budut ispol'zovat' ego. V sootvetstvii s nim bol'shinstvo proizvoditelej
robotov predlagaet vozmozhnost' zashchitit' Veb-servery ot nezhelatel'nogo poseshcheniya
ih poiskovymi robotami.</P>

<P align=justify>Poslednyuyu versiyu etogo dokumenta mozhno najti po adresu
<A HREF="http://info.webcrawler.com/mak/projects/robots/robots.html">info.webcrawler.com/mak/projects/robots/robots.html
</A></P>

<P><A NAME="2"></A><B>Vvedenie</B></P>

<P align=justify>Poiskovye roboty (wanderers, spiders) - eto programmy,
kotorye indeksiruyut veb-stranicy v seti Internet.</P>

<P align=justify>V 1993 i 1994 godah vyyasnilos', chto indeksirovanie robotami
serverov poroj proishodit protiv zhelaniya vladel'cev etih serverov. V chastnosti,
inogda rabota robotov zatrudnyaet rabotu s serverom obychnyh pol'zovatelej,
inogda odni i te zhe fajly indeksiruyutsya neskol'ko raz. V drugih sluchayah
roboty indeksiruyut ne to, chto nado, naprimer, ochen' &quot;glubokie&quot;
virtual'nye direktorii, vremennuyu informaciyu ili CGI-skripty. |tot standart
prizvan reshit' podobnye problemy.</P>

<P><A NAME="3"></A><B>Naznachenie</B></P>

<P align=justify>Dlya togo, chtoby isklyuchit' poseshchenie servera ili ego chastej
robotom neobhodimo sozdat' na servere fajl, soderzhashchij informaciyu dlya upravleniya
povedeniem poiskovogo robota. |tot fajl dolzhen byt' dostupen po protokolu
HTTP po lokal'nomu URL /robots.txt. Soderzhanie etogo fajla sm. nizhe. </P>

<P align=justify>Takoe reshenie bylo prinyato dlya togo, chtoby poiskovyj robot
mog najti pravila, opisyvayushchie trebuemye ot nego dejstviya, vsego lish' prostym
zaprosom odnogo fajla. Krome togo fajl /robots.txt legko sozdat' na lyubom
iz sushchestvuyushchih Veb-serverov.</FONT></P>

<P>Vybor imenno takogo URL motivirovan neskol'kimi kriteriyami:
</P>

<UL>
<LI>Imya fajla dolzhno bylo byt' odinakovym dlya lyuboj operacionnoj
sistemy</LI>

<LI>Rasshirenie dlya etogo fajlya ne dolzhno bylo trebovat' kakoj-libo
perekonfiguracii servera</LI>

<LI>Imya fajla dolzhno bylo byt' legko zapominayushchimsya i otrazhat'
ego naznachenie</LI>

<LI>Veroyatnost' sovpadeniya s sushchestvuyushchimi fajlami dolzhna
byla byt' minimal'noj</LI>
</UL>

<P><A NAME="4"></A><B>Format</B></P>

<P align=justify>Format i semantika fajla /robots.txt sleduyushchie: </P>

<P align=justify>Fajl dolzhen soderzhat' odnu ili neskol'ko zapisej (records),
razdelennyh odnoj ili neskol'kimi pustymi strokami (okanchivayushchimisya CR,
CR/NL ili NL). Kazhdaya zapis' dolzhna soderzhat' stroki (lines) v forme: </P>

<P>&quot;&lt;field&gt;:&lt;optional_space&gt;&lt;value&gt;&lt;optional_space&gt;&quot;.
</P>

<P>Pole &lt;field&gt; yavlyaetsya registronezavisimym. </P>

<P align=justify>Kommentarii mogut byt' vklyucheny v fajl v obychnoj dlya UNIX
forme: simvol # oznachaet nachalo kommentariya, konec stroki - konec kommentariya.
</P>

<P>Zapis' dolzhna nachinat'sya s odnoj ili neskol'kih strok
User-Agent, sledom dolzhna byt' odna ili neskol'ko strok Disallow, format
kotoryh priveden nizhe. Neraspoznannye stroki ignoriruyutsya.</P>

<P>User-Agent</P>

<UL>
<LI>znacheniem &lt;value&gt; etogo polya dolzhno yavlyat'sya imya
poiskovogo robota, kotoromu v etoj zapisi ustanavlivayutsya prava dostupa.
</LI>

<LI>esli v zapisi ukazano bolee odnogo imeni robota, to prava
dostupa rasprostranyayutsya dlya vseh ukazannyh imen.</LI>

<LI>zaglavnye ili strochnye simvoly roli ne igrayut </LI>

<LI>esli v kachestve znach