Oglavlenie fajla ( /LYNX/WEBMASTER/robots-faq.txt )



  • The Web Robots FAQ

  • Neskol'ko slov o tom, kak rabotayut roboty (spiders) poiskovyh mashin

  • Standart isklyuchenij dlya robotov
    • Size: 43399 bytes
    Last-modified: Thu, 28 May 1998 14:14:26 GMT

    I kusochkami po 40Kb: [1][2]