Good question. There is not currently an option for this, but you can implement it yourself by modifying the qa_question_validate(...) function in qa-app-post-create.php. You would add something like this inside:
if (count(qa_tagstring_to_tags($tagstring))>5)
$errors['tags']='A maximum of 5 tags are allowed';
(code not tested, so apologies for any bugs...)