Since every page has the "canonical" meta tag, a link to domain.com/123/ would be treated the almost same as a link to domain.com/123/question-title, so there wouldn't be much of a difference.
If possible, you should get links to the canonical URL, but it's not essential.