Question

Should I allow google to Crawl everything ?

Answer 1

Sure, if your site is open to the public I don't see why not.

If you have any content you're not certain if it should or should not be indexed, you can block search indexing with 'noindex' 

Let's say you don't feel like sharing profile pages because it has too sensitive information about your users. To prevent all bots from indexing your User pages, you would open your  qa-theme.php  file and add the following code.

public function head_metas()
{
    if ($this->template === 'user') {
        $this->output('<meta name="robots" content="noindex">');
    }
    parent::head_metas();
}

This if you don't have a  head_metas()  function in your  qa-theme  already of course. If you have, you'd just paste that output in. If you only want to block Google from indexing, then you'd do:

<meta name="googlebot" content="noindex">

Read more about indexing here: https://support.google.com/webmasters/answer/93710?hl=en

Comments

Thanks..But should I use robots.txt to stop crawling of pages because I have seen in SEO video that google bot come on site and crawl for some limited time. So, if googlebot crawls not useful pages like users, feed, tag, etc. So I think robots.txt is better than noindex.
see this robots.txt
User-agent: *

Disallow: /*/login

Disallow: /*?qa-rewrite=*

Disallow: /*/forgot

Disallow: /*/register

Disallow: /*/questions/*?sort=views&start=*

Disallow: /*/questions?sort

Disallow: /*/chat

Disallow: /*/activity/*

Disallow: /*/cdn-cgi/*

Disallow: /*/questions?start=*

Disallow: /*/questions?sort=*

Disallow: /*/search?q=*

Disallow: /search/*

Disallow: /?s=*

Disallow: /search/?query=*

Disallow: /ip/*

Disallow: /login

Disallow: /ask

Disallow: /forgot

Disallow: /register

Disallow: /admin

Disallow: /feed

Disallow: /tag

Disallow: /tags

Disallow: /users

Disallow: /user

Disallow: /message/

# allow google image bot to search all images

User-agent: Googlebot-Image

Allow: /*

Sitemap: https://yourdomain.com/sitemap.xml

---

On the "read more" link I posted, there's a note that says "important," at the very beginning of the page, that is exactly talking about this.

https://support.google.com/webmasters/answer/93710?hl=en