Pues este es mi robots.txt:
Código:
User-agent: *
Disallow: /cgi-bin/
Disallow: /forum/admin/
Disallow: /forum/cache/
Disallow: /forum/db/
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/language/
Disallow: /forum/templates/
Disallow: /forum/faq.php
Disallow: /forum/groupcp.php
Disallow: /forum/login.php
Disallow: /forum/memberlist.php
Disallow: /forum/modcp.php
Disallow: /forum/posting.php
Disallow: /forum/privmsg.php
Disallow: /forum/profile.php
Disallow: /forum/search.php
Disallow: /forum/viewonline.php
User-agent: Googlebot
# disallow files ending with these extensions
Disallow: /*.inc$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
# disallow all files with? in url
Disallow: *mark=*
Disallow: *view=*
# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow:
Allow: /*
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
Allow: /*
User-agent: *
Disallow: /stats
Disallow: /dh_
Disallow: /V
Disallow: /z/j/
Disallow: /z/c/
Disallow: /cgi-bin/
Disallow: /viewtopic.php
Disallow: /viewforum.php
Disallow: /index.php?
Disallow: /posting.php
Disallow: /groupcp.php
Disallow: /search.php
Disallow: /login.php
Disallow: /post
Disallow: /member
Disallow: /profile.php
Disallow: /memberlist.php
Disallow: /faq.php
Disallow: /templates/
Disallow: /mx_
Disallow: /db/
Disallow: /admin/
Disallow: /cache/
Disallow: /images/
Disallow: /includes/
Disallow: /common.php
Disallow: /index.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /privmsg.php
Disallow: /viewonline.php
Disallow: /images/
Disallow: /rss.php
User-agent: Googlebot
# disallow all files ending with these extensions
Allow: /sitemap.xml
Disallow: /*.php$
Allow: /sitemap.xml
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
# disallow all files with? in url
Disallow: /*?*
Disallow: /*?
# disallow all files in /wp- directorys
Disallow: /wp-*/
# disallow archiving site
User-agent: ia_archiver
Disallow: /
# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow:
Allow: /*.gif$
Allow: /*.png$
Allow: /*.jpeg$
Allow: /*.jpg$
Allow: /*.ico$
Allow: /*.jpg$
Allow: /images
Allow: /z/i/
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Allow: /*
creo que en lo que se refiere a los robots todo esta en orden, tambien ya mande mi sitemap a google, talvez me toque esperar, y bueno viendo bien ya el dia de hoy se indexaron 3 paginas de mi foro...
Una pregunta mas... en el .htaaccess del root de mi web tengo esto:
Código:
Redirect 301 /index.php [direccion de mi web]/ forum
ErrorDocument 404 /404.htm
y en la carpeta /forum/ tengo lo siguiente:
Código:
DirectoryIndex index.html index.htm portal.php index.php
Solo es una pequeña duda, mas que nada del .htaaccess del root de mi web, lo de redirect 301 quisiera saber si afecta de alguna manera, o hay algun otro codigo para redireccionar del root a la carpeta /forum/ ?