If you are using any of the built-in themes then the columns setting no longer has any effect. It was only for the table layout of users/tags.
You can change the columns now by updating the CSS. In qa-styles.css look for the classes .qa-top-tags-item or .qa-top-users-item and change the widths accordingly (smaller width = more columns)