Google Bot causing thousands of search events (event logger)

Question

Just checked table qa_eventlog and found dozens of SEARCH events, triggered by IP 66.249.78.148 which seems to be Google Search bot. crawl-66-249-78-148.googlebot.com

As this is not human, I do not want to have those events listed in my database (and who knows how enormous the data grows if bots visited my forum each day).

Any counter measure (but blocking the IP which is undesired in this case...)

Thanks,
Kai

Comments

And by the way, it is triggering the same search query all the time, how strange: "searched for" "was ist mit lim n ..?q=was ist mit lim..."
There is another ?q parameter inside the search string :?(

---

I checked the IP and its connected events:
21,611 total queries (!)

---

For now, I switched logging of search events off, see how-to here: http://www.question2answer.org/qa/22480/any-way-to-exclude-search-events-from-event-logger

Removing all "search" events from my database saved me 5 MB (!)

Answer 1

You can block google bot in your robots.txt

Comments

My feeling tells me that it has to do with the core hack of page-not-found.php suggested here: http://www.question2answer.org/qa/21876/trick-redirect-error-404-to-search-page

There the words from the URL are set into the search field, that would explaing the ?q parameter somehow...

---

You can block search bots from web pages you don't want them to look at. So I'm going to do it to you again. Read up on Robot.txt blocking webpages from search bots. With this option you don't have to block the whole of the website.

---

of course, thank you, I overlooked the specific folder blocking...

---

All right, I am using now in robots.txt:

User-agent: *
Disallow: /search/
Disallow: /6242/

where /6242/ has been the post that caused the google loop!

---
And just now I see that the URL for the google loop has a "%3F" inside which hits my recent redirect loop problem: http://www.question2answer.org/qa/22544/redirecting-problem-redirect-loop-when-instead-question-mark ♫ that was the reason!