Hola...
tengo una duda sobre los archibos robots.txt, ya que he estado leyendo y tenia entendido que esta expresion " Disallow: /carpeta/ejemplo.php " no permite a los crawels ver el contenido de las mismas pero cuando genero el archibo robots.txt ( con Prestashop ) me genera un robpts.txt automáticamente, me lo genera asi:
# robots.txt automaticaly generated by PrestaShop e-commerce open-source solution
# http://www.prestashop.com - http://www.prestashop.com/forums
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/wc/robots.html
# GoogleBot specific
User-agent: Googlebot
Disallow: /carpeta/*orderby=
Disallow: /carpeta/*orderway=
Disallow: /carpeta/*tag=
Disallow: /carpeta/*id_currency=
Disallow: /carpeta/*search_query=
Disallow: /carpeta/*id_lang=
Disallow: /carpeta/*back=
Disallow: /carpeta/*utm_source=
Disallow: /carpeta/*utm_medium=
Disallow: /carpeta/*utm_campaign=
Disallow: /carpeta/*n=
# All bots
User-agent: *
# Directories
Disallow: /carpeta/classes/
Disallow: /carpeta/config/
Disallow: /carpeta/download/
Disallow: /carpeta/mails/
Disallow: /carpeta/modules/
Disallow: /carpeta/translations/
Disallow: /carpeta/tools/
# Files
Disallow: /carpeta/addresses.php
Disallow: /carpeta/address.php
Disallow: /carpeta/authentication.php
Disallow: /carpeta/cart.php
Disallow: /carpeta/discount.php
Disallow: /carpeta/footer.php
Disallow: /carpeta/get-file.php
Disallow: /carpeta/header.php
Disallow: /carpeta/history.php
Disallow: /carpeta/identity.php
Disallow: /carpeta/images.inc.php
Disallow: /carpeta/init.php
Disallow: /carpeta/my-account.php
Disallow: /carpeta/order.php
Disallow: /carpeta/order-opc.php
Disallow: /carpeta/order-slip.php
Disallow: /carpeta/order-detail.php
Disallow: /carpeta/order-follow.php
Disallow: /carpeta/order-return.php
Disallow: /carpeta/order-confirmation.php
Disallow: /carpeta/pagination.php
Disallow: /carpeta/password.php
Disallow: /carpeta/pdf-invoice.php
Disallow: /carpeta/pdf-order-return.php
Disallow: /carpeta/pdf-order-slip.php
Disallow: /carpeta/product-sort.php
Disallow: /carpeta/search.php
Disallow: /carpeta/statistics.php
Disallow: /carpeta/attachment.php
Disallow: /carpeta/guest-tracking.php
# Sitemap
Sitemap: http://www.miweb.es/sitemap.xml
mi pregunta es.... verdad que me esta cortando el acceso de los crawels a todos estos archivos o carpetas relacionada ???
