Hi, thanks for the question. It's a complex situation because most of the elements are there to aid theme/plugin customization. For example, the *-pad classes allow you to style prefixes/suffixes differently like the "N votes" where the number is bigger. So most of them are necessary.
The *-clear classes are a legacy for older browsers that didn't support clearfix hacks. The newer themes don't use them so we can probably update the older themes to use clearfix and then remove the elements.
The elements don't affect layout shift either since the HTML is sent from the server, not loaded on-the-fly with Javascript. I get a CLS of 0.02 on this site, and 0 on my own site. (I think the 0.02 is due to the dropdown banner that appears at the top of the page, which you can turn off in settings, and users can close themselves.)
Also when I test in Google's PageSpeed Insights, the DOM size warning is under a Diagnostics sections that it says doesn't affect the performance score. The pages I tested had scores over 95 for desktop, although a bit lower for mobile.
If you have any more ideas on which elements to remove I'd be happy to discuss them.