Question

What is the best robots.txt for SEO ?

Answer 1

User-agent: *

Disallow: /*/login

Disallow: /*?qa-rewrite=*

Disallow: /*/forgot

Disallow: /*/register

Disallow: /*/questions/*?sort=views&start=*

Disallow: /*/questions?sort

Disallow: /*/chat

Disallow: /*/activity/*

Disallow: /*/cdn-cgi/*

Disallow: /*/questions?start=*

Disallow: /*/questions?sort=*

Disallow: /*/search?q=*

Disallow: /search/*

Disallow: /?s=*

Disallow: /search/?query=*

Disallow: /ip/*

Disallow: /login

Disallow: /ask

Disallow: /forgot

Disallow: /register

Disallow: /admin

Disallow: /feed

Disallow: /tag

Disallow: /tags

Disallow: /users

Disallow: /user

Disallow: /message/

# allow google image bot to search all images

User-agent: Googlebot-Image

Allow: /*

Sitemap: https://yourdomain.com/sitemap.xml

correct me if I am wrong and what should else I add ? 

Comments

A lot of that you don't need to block. The tag pages are useful for indexing based on certain words, similar for users.

The admin, IP and message pages are not visible to logged-out users and aren't linked from anywhere public. (If anything, adding them to your robots.txt might be considered insecure as it tells a hacker those pages exist.)

I don't know if login/register are even worth it to block. The only reason I blocked them on my site originally was to prevent them coming up higher in search results when my site was new. Probably not needed now.