robots.txt for SEO

Question

What will be appropriate robots.txt for question2answer sites?

Answer 1

I use the following, which blocks the login page and forgot password page (because they usually contain the "?to=" parameter so you get duplicate content).

User-agent: *
Disallow: /qa/login
Disallow: /qa/forgot

"qa/" here is the subfolder you have Q2A in - if you're using a root domain remove that bit.

Comments

Thanks for quick answer.

how about this -

User-agent: *

Disallow: /login*
Disallow: /register*
Disallow: /forgot

Sitemap: http://www.xxxxxx.com/sitemap.xml

---

You don't need the * after login and register. Using `Disallow: /qa/login` blocks everything beginning with that URL. Also I'm not sure if a space is allowed between the user-agent and disallow lines.

---

Adding only this two tags will resolve duplicate issue or do I need to add some more?

Answer 2

User-agent: *
Disallow: /login
Disallow: /index.php?qa-rewrite=
Disallow: /ask
Disallow: /forgot
Disallow: /register
Disallow: /questions?sort
Disallow: /chat
Disallow: /admin
Disallow: /activity/*
Disallow: /search?q=
Disallow: /cdn-cgi/
# allow google image bot to search all images
User-agent: Googlebot-Image


Allow: /*


Sitemap: Sitemap address

Comments

Why block the users pages? And why /questions/*, that's the list of questions which you would want indexed.

---

Thank you
I remove it

---

How about blocking
1. Feedback page (To avoid thin content)
2. Tags (To avoid thin content)
3. Terms and Condition (To avoid duplicate Content)
4. Privacy Policy (To avoid duplicate content)
+Please Add More

---

@Scott

http://www.question2answer.org/qa/56477/google-recognizes-q2a-category-and-tag-pages-as-duplicate