eniya etogo polya ukazan simvol "*",
to zadannye v etoj zapisi prava dostupa rasprostranyayutsya na lyubyh poiskovyh
robotov, zaprosivshih fajl /robots.txt</LI>
</UL>

<P>Disallow</P>

<UL>
<LI>znacheniem &lt;value&gt; etogo polya dolzhen yavlyat'sya chastichnyj
URL, kotoryj ne dolzhen indeksirovat'sya. |to mozhet byt' polnyj put' ili
chastichnyj; lyuboj URL, nachinayushchijsya s takogo puti ne dolzhen indeksirovat'sya.
Naprimer, Disallow: /help zakryvaet i /help.html, i /help/index.html, togda
kak <BR>
Disallow: /help/- tol'ko /help/index.html. </LI>

<LI>esli znachenie Disallow ne ukazano, to eto oznachaet, chto
indeksiruetsya vse derevo katalogov servera</LI>
</UL>

<P align=justify>Lyubaya zapis' (record) dolzhna sostoyat' hotya by iz odnoj
stroki (line) User-Agent i odnoj - Disallow</P>

<P align=justify>Esli fajl /robots.txt pust, ili ne otvechaet zadannomu
formatu i semantike, ili ego ne sushchestvuet, lyuboj poiskovyj robot budet
rabotat' po svoemu algoritmu.</P>

<P><A NAME="5"></A><B>Primery</B></P>

<P>Primer 1: </P>

<PRE># robots.txt for http://www.site.com

        User-Agent: *

        Disallow: /cyberworld/map/ # this is an infinite virtual URL space
        Disallow: /tmp/ # these will soon disappear
</PRE>

<P align=justify>V primere 1 zakryvaetsya ot indeksacii soderzhimoe direktorij
/cyberworld/map/ i /tmp/. </P>

<P>Primer 2: </P>

<PRE># robots.txt for http://www.site.com

        User-Agent: *

        Disallow: /cyberworld/map/ # this is an infinite virtual URL space

# Cybermapper knows where to go

        User-Agent: cybermapper

        Disallow:
</PRE>

<P align=justify>V primere 2 zakryvaetsya ot indeksacii soderzhimoe direktorii
/cyberworld/map/, odnako poiskovomu robotu cybermapper vse razresheno. </P>

<P>Primer 3:</P>

<PRE># robots.txt for http://www.site.com

        User-Agent: *

        Disallow: /
</PRE>

<P align=justify>V primere 3 lyubomu poiskovomu robotu zapreshchaetsya indeksirovat'
server. </P>

<P><A NAME="6"></A><B>Primechaniya perevodchika</B> </P>

<P align=justify>V nastoyashchee vremya standart neskol'ko izmenilsya, naprimer,
mozhno zapisyvat' v stroke User-Agent neskol'ko imen robotov, razdelennyh
probelami ili tabulyatorami.</P>

<P><A NAME="7"></A><B>Adresa avtorov</B></P>

<P>Martijn Koster, <A HREF="mailto:m.koster@webcrawler.com">m.koster@webcrawler.com</A>
</FONT></P>

<P>Perevod: Andrej Alikberov, <A HREF="mailto:info@citmgu.ru?Subject='From CIT-FORUM.COM'">info@citmgu.ru</A>
</FONT></P>