que tal maestros del SEO
solo pedirles un favorsote, pueden echarle un ojo a mi robots.txt
para saber si esta bien, si debo agregar/quitar/cambiar algo
por favor
gracias
robots.txt
Código:
User-agent: *
Allow: /
Disallow:
Disallow: /access/
Disallow: /administracion/
Disallow: /ckeditor/
Disallow: /conexion_mysql/
Disallow: /css/
Disallow: /js/
Disallow: /source/
Sitemap: http://www.buzonxalapa.com/sitemap.xml
User-agent: Googlebot
# disallow all files ending with these extensions
Disallow: /access/
Disallow: /administracion/
Disallow: /ckeditor/
Disallow: /conexion_mysql/
Disallow: /css/
Disallow: /js/
Disallow: /source/
User-agent: Googlebot-Image
Disallow:
Allow: /*
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Allow: /
Disallow:
Disallow: /access/
Disallow: /administracion/
Disallow: /ckeditor/
Disallow: /conexion_mysql/
Disallow: /css/
Disallow: /js/
Disallow: /source/
User-agent: Slurp
Allow: /
Disallow:
Disallow: /access/
Disallow: /administracion/
Disallow: /ckeditor/
Disallow: /conexion_mysql/
Disallow: /css/
Disallow: /js/
Disallow: /source/
User-agent: Scooter
Allow: /
Disallow:
Disallow: /access/
Disallow: /administracion/
Disallow: /ckeditor/
Disallow: /conexion_mysql/
Disallow: /css/
Disallow: /js/
Disallow: /source/
User-agent: duggmirror
Disallow: /
User-agent: wget
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: libwww
Disallow: /
User-agent: noxtrumbot
Crawl-delay: 30
User-agent: msnbot
Crawl-delay: 30
User-agent: Slurp
Crawl-delay: 30