Using robots.txt with addon domains

[mrongey]

I want to clarify my understanding of using robots.txt with addon domains to avoid duplicate content.

This is my current understanding:

Let's say I have www.domain.com and www.addon.com is an addon in the same account. If I put a robots.txt in the root directory of www.domain.com and disallow access to /addon, the spider crawling www.domain.com will not go into that folder. However, when a spider crawls www.addon.com it will not have a problem because it does not see the robots.txt file.

Is this correct?

[slapshotw]

Yes, it is. But it may actually be smarter not to include it. Making that robots.txt will actually be the only thing alerting search engines that the folder exists. Many unscrupulous search engine algorithms and spam bots will read the robots.txt, see the existence of the folder, and then ignore your instructions.

As long as you turn off directory indexing and don't link to the addon site using the folder hierarchy, I don't see how any search engine would ever know it exists.

In addition, to fix any other chance of problems, I'd stick a redirect command in the addon's public_html. Make sure you do a 301 permanent redirect, and have it redirect from domain.com/addon to addon.com, with file level redirection. You can do this from cPanel or search on this forum for the proper commands.

[charlesgan]

Originally Posted by mrongey

I want to clarify my understanding of using robots.txt with addon domains to avoid duplicate content.

This is my current understanding:

Let's say I have www.domain.com and www.addon.com is an addon in the same account. If I put a robots.txt in the root directory of www.domain.com and disallow access to /addon, the spider crawling www.domain.com will not go into that folder. However, when a spider crawls www.addon.com it will not have a problem because it does not see the robots.txt file.

Is this correct?

ya correct. AS the addon.com is another website, and its a big door open for spider.

[quietFinn]

Originally Posted by charlesgan

ya correct. AS the addon.com is another website, and its a big door open for spider.

Where is that big door?

[charlesgan]

domain.com is main door. You can access primary domain and also addon domain.

And for the addondomain.com is another door directly to the addon domain.

its domain pointer thing in the cpanel that allow the multiple addon domain. So, it create this confusion. Use .htaccess to block it will one the solution. or you can get a reseller account, each domain are in own cpanel

[quietFinn]

Originally Posted by charlesgan

domain.com is main door. You can access primary domain and also addon domain.

The directory where the add-on domain resides is not visible, i.e. you must know the directory name to be able to browse to it.
Search engine bots can't see anything more than a normal browser.

[Break the Silence]

Originally Posted by slapshotw

Many unscrupulous search engine algorithms and spam bots will read the robots.txt, see the existence of the folder, and then ignore your instructions.

Quite correct...

In a case like this, it would be unwise to rely strictly on robots.txt. The safer option, (as stated previously), would be to simply avoid a direct downward link between the two folders.