Yep, it's just taking the information from ^userpoints, via qa_db_user_points_selectspec():
'source' => '^userpoints WHERE userid='.($isuserid ? '$' : '(SELECT userid FROM ^users WHERE handle=$ LIMIT 1)'),
I'm not sure this is necessarily a bug... just because an answer has been hidden, doesn't mean you didn't give it :)
A good question might be, if the answer is deleted, does the count change?