Robots.txt and subdomains; other newbie questions

theidolguy · #1 03-26-2008, 10:18 AM

I'm new to the whole web admin thing, so please forgive my ignorance. How do I set up robots.txt to stop search engine bots to not index a subdomain? Do I put the following robots.txt file in my www folder:

Code:

User-agent: *
Disallow: /my_subdomain/

And/or do I need to do this for robots.txt in the folder of my subdomain?

Code:

User-agent: *
Disallow: /

One more unrelated question - what kind of protection should I put in a WordPress install to protect the lower level directories like wp-admin, wp-includes, and wp-content? Should I or should I not turn off indexing in the Index Manager? What about an SMF install - should I turn off indexing in all the subdirectories, too?

esl · #2 03-26-2008, 10:30 AM

I don't know myself, but you might be able to find your answer here:
http://www.robotstxt.org/

kompreszor · #3 03-26-2008, 04:53 PM

For the robot.txt I do it just like you have it, only I use them both, one in the root and the other in the sub. If a robot hits the root (main domain ) and sees the /subdomain then the robots.txt there should cover it. If it come through sub.domain.tld then it will see the robots.txt in the subdomain folder and read it as this would be the root for that domain. Of course this only applies to robots that obey the robots.txt file :-)

I always turn off indexes if I don't need them. Just add "Options -Indexes" without the quotes to you htaccess file in the public_html directory it will then apply to all sub directories.

I don't use WP but a quick google found this;
http://www.askapache.com/wordpress/h...d-protect.html