best robots.txt file

Question

Hello everyone, i only want to index topics, comments and tags in my website. What kind of robot.txt should i add for this? or could you give me a ready robot.txt file? Thanks..

Answer 1

Why not use an existing one, like  that of Stack Exchange's?  

http://physics.stackexchange.com/robots.txt

 

Comments

Is this from a Q2A website?  I want to restrict access to this subdirectory:
register?to=

However this is not included there.

Answer 2

You can use this file.

https://meta.question2answer.info/robots.txt
(change sitemap url)
keep monitoring all unwanted bots which are slowing down your site and then add to robots.txt list. It totally depends on your site. check server log /var/log/httpd/access_log file for bot info. Dont block useful bots like Google, Bing etc.

Comments

Can you suggest a robot.txt for q2a integrated with wordpress?

---

You can use this one https://question2answer.info/robots.txt

If you are having q2a and wordpress on same domain then combine these two robots.txt file.