Promoting a single one of your domains on the search engines
Archive - Originally posted on "The Horse's Mouth" - 2012-01-22 22:59:02 - Graham Ellis
We have a number of different domains offering information about our courses, and technical content too. There's www.wellho.net which is our main, worldwide site. But then there's www.wellho.co.uk which carries information with a U.K. bias and mobile.wellho.net which presents for the smaller screen size of a mobile phone.
Do we want all of these domains listed by the search engines - a wide spread to catch as many potential customers as possible? Not really, because whilst we might get a higher number of pages indexed if we advertise them all, we'll also have their ranking diluted if there's duplication of content. Much better to have one domain ranked really well than have our ranking spread out between many, so that none appears as highly as we would like.
In database design, you learn that "source data should only be stored once" and indeed we have a single server (but with backup, of course) where our definitive editable content is stored, and we use virtual hosts and host forwarding to answer all browser requests from there. One of the files stored there is /robots.txt which is the file that well behaved robots read in order to establish where they are welcome (and where excluded) on a site. See [here] for more about the format of robots.txt, a list of robots, etc.
In order to ensure that www.wellho.net is indexed, but www.wellho.co.uk, mobile.wellho.net and others are not, our robits.txt page actually runs a PHP script which sends a strong "DO NOT INDEX" message for our subsidiary domains, and a much more welcoming, and carefully tuned, response for www.wellho.net.
Here's how we do it:
<?php
header ("Content-type: text/plain");
print ("# robots.txt file for $_SERVER[HTTP_HOST]\n");
$mainsite = 1;
# If forwarded, this isn't the main site
if ($_SERVER[HTTP_X_FORWARDED_HOST]) $mainsite = 0;
if ($mainsite == 0)
print ("# robots.txt file from $_SERVER[HTTP_X_FORWARDED_HOST]\n");
# If not a www. domain, this isn't the main site
if ( ! preg_match("/^www/",$_SERVER[HTTP_HOST])) $mainsite = 0;
if ( $mainsite == 0) {
?>
# This is not the main site - don't index it to avoid dilution
# Please refer instead to www.wellho.net
User-agent: *
Disallow: /
<?php } else {
# This is what we tell robots for the main site
?>
Why different content for a laptop browser and a phone? Here is the same blog article, as presented on a regular web page and for an ipad or mobile device:
