Tengo el siguiente archivo robots:
Código:
<!-- BEGIN Developer Shed™ Network's Robots.txt Generator -->
<!-- Copyright 2001-2007 Developer Shed™, LLC All Rights Reserved -->
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td valign="top">
<table width="300" border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="16" height="16"><img src="http://images.devshed.com/sc/seotools/image_01.gif" width="16" height="16"></td>
<td width="130" height="16" background="http://images.devshed.com/sc/seotools/bckg06.gif"><img src="http://images.devshed.com/sc/seotools/spacer.gif" width="130" height="16"></td>
<td width="118" height="16" background="http://images.devshed.com/sc/seotools/bckg06.gif"><img src="http://images.devshed.com/sc/seotools/spacer.gif" width="150" height="16"></td>
<td width="40" background="http://images.devshed.com/sc/seotools/bckg06.gif"><img src="http://images.devshed.com/sc/seotools/spacer.gif" width="40" height="16"></td>
<td width="18" height="16"><img src="http://images.devshed.com/sc/seotools/image_02.gif" width="16" height="16"></td>
</tr>
<tr>
<td height="30" background="http://images.devshed.com/sc/seotools/bckg07.gif"> </td>
<td class=toolbox width="*" bgcolor="#CCCCCC" colspan=2 valign=top>
<B><font size=+1>Robots.txt Generator Tool</font></b> © <a href="http://www.seochat.com">SEO Chat™</a>
<font size="2" face="Courier New, Courier, mono">
<form action="http://www.seochat.com/" method="GET" name="form"><input type=hidden name=go value=1><input tabindex=1 type="hidden" name="option" value="com_seotools"><input tabindex=2 type="hidden" name="tool" value="21"><p><b>Allowed User Agent</b><br>Select user agent or use default for all agents<br><select tabindex=3 name="usera" id="usera" size=""><option value="*" >All Agents</option><option value="ia_archiver" >Alexa Agent</option><option value="Scooter" >AltaVista Agent</option><option value="ArchitextSpider" >AOL Netfind/Excite Agent</option><option value="Googlebot" >Google Agent</option><option value="Slurp" >HotBot Agent</option><option value="InfoSeek Sidewinder" >InfoSeek Agent</option><option value="Lycos_Spider_(T-Rex)" >Lycos Agent</option><option value="MSNBot" >MSN Search Crawler</option><option value="Gulliver" >Northern Lights Agent</option><option value="WebCrawler/3.0" >Webcrawler Agent</option></select><p><b>Enable All Webpages</b><br><input tabindex=4 type="radio" name="allpages" id="allpages" value="1" > Yes<br><input tabindex=5 type="radio" name="allpages" id="allpages" value="0" checked="checked"> No<br><p><b>Exclude These URL's</b><br>Enter URLs you wish to exclude<br><textarea tabindex=6 name="exclude_list" cols=30 rows=5>/cgi-bin/
/images/</textarea><p><b>Enter Captcha To Continue</b><br>To prevent spamming, please enter in the numbers and letters in the box below<br><input type=text tabindex=7 name="imageverify" size=32 value=""><input tabindex=8 type="hidden" name="timehsh" value="vb7KxMA="><p><img src=http://www.seochat.com/includes/showcaptcha.php?bd=rb22ysY= border=1><p><input tabindex=9 type=submit name="toolsubmit" id="toolsubmit" value="Create"></form>
</font>
</td>
<td width="40" valign="top" bgcolor="#CCCCCC"></td>
<td width="18" background="http://images.devshed.com/sc/seotools/bckg03.gif"> </td>
</tr>
<tr>
<td width="16"><img src="http://images.devshed.com/sc/seotools/image_03.gif" width="16" height="16"></td>
<td colspan="3" width="*" height="16" background="http://images.devshed.com/sc/seotools/bckg04.gif"><img src="http://images.devshed.com/sc/seotools/spacer.gif" width="16" height="16"></td>
<td width="18"><img src="http://images.devshed.com/sc/seotools/image_04.gif" width="16" height="16"></td>
</tr>
</table>
</td>
</tr>
</table>
<!-- END Developer Shed™ Network's Robots.txt Generator -->
El nombre es Robots.txt pero no se que significa ese codigo ni que tengo que hacer con él.