The tool you mention is for front-end performance, whereas the caching system in Q2A is all about the back-end performance*. Some of the page speed tools may show a "Time To First Byte" (TTFB) measurement, which Q2A's caching system will improve.
How much the caching helps will depend mainly on how may anonymous (not logged in) visitors you get. Currently, caching question pages only works for anonymous visitors (because the queries for that page mix in user details). Other queries such as the related questions widget are cached for all visitors. I tested on my site and less than 5% of my visitors were logged in, but as I have a high-traffic site getting visitors from Google, this may not be typical of smaller sites.
Caching effectiveness also depends on how long you set the cache for. If you set it too short there is no point as it will have expired by the time the next person comes along. You can do a bit of maths to decide how long you should cache for. Let's say your most popular question gets about 100 visitors per day. 1 day = 1440 minutes so that means on average, someone visits that question once every 14.4 minutes. And that's just an average, so quiet periods on your site could have 1 visitor per hour or less. To get the benefit you'd need to cache a page for several hours.
Hope that helps.
Edit: regarding your question about "frequently changing" sites, that in itself doesn't matter. Firstly there is an option to prevent caching of new posts (because they are more likely to be updated soon). Secondly, updating an older question doesn't purge the cache - anonymous visitors will still see the cached question until it expires.
I also forgot to mention Memcached, which is an alternate caching system using memory instead of files on disk. However, to use it you need to install the Memcached service, and enable the Memcached PHP extension. But it should be much more efficient than the default file cache. I haven't got around to testing on my live site yet but I'll let you know how it goes.
* While the caching system only works on the back-end, there have been some improvements to the front-end in v1.8 as well:
- The JavaScript has been combined into one file, qa-global.js. That means it is loaded once and can be cached on all future pages.
- A lot of JS code duplication has been streamlined.
- There have been various improvements to the SnowFlat CSS since it was first added to Q2A (some of that was already done in 1.7.x, I don't remember exactly how much was done in 1.8).
- There's a new option in admin, "Minify HTML" which makes the HTML slightly smaller. This is actually both a back-end and front-end improvement as the PHP code outputting the HTML is faster too.
