Let’s say my domain is example.com. On www.example.com, I had set up my main website (set up with Blogger) and used go.example.com for URL shortening (setup with GoDaddy Shortened Service). Now, I want to block all go.example.com URLs so that they can't be indexed.
If I use rel="noindex,nofollow" for go.example.com URLs in my main blog, then, does this affect my blog search engine optimization?
With this robots.txt on go.example.com, you disallow conforming bots to crawl any URL from that host:
# https://go.example.com/
User-agent: *
Disallow: /
If you are fine with crawling, but you want to prevent indexing (you can’t have both), you have to use noindex (either in the meta-robots element or in the HTTP header X-Robots-Tag). Note that you can’t use noindex as link type in the rel attribute.
That said, if you always redirect (e.g., with 301) from go.example.com to the canonical URL (on another host), search engine bots have no reason to index your document on go.example.com, as you don’t provide any content, only the redirect.
