OK I think I found the problem - the times correspond with when I approve or reject a post. I recently added moderation and I've only noticed the problem recently, so that fits.
If it matters, the moderation is done via a custom filter plugin I wrote, not the options in Admin > Spam. The plugin sends answers for moderation if the question is more than a month old and the user has less than a certain number of points.