This is a theme issue, and with the default Q2A themes only occurs on pages with many answers. There's not much that can be done to avoid that, if you have a lot of answers and comments you're going to have a lot of HTML elements on the page.
You could try reducing the number of answers per page, and how many comments are shown below each answer (the options "Partially hide comments if more than" and "If partially hidden, show most recent" on Admin > Viewing).
On the whole it's not worth worrying about, it doesn't affect page load that much.