In theory, yes it's possible to use another database. It's not supported out of the box but you can override the database functions to use another DB engine.
Take a look at the file qa-include/qa-db.php and the docs for override plugins. You will need to override most of the functions there to use MSSQL functions for connecting etc. You may also need overrides for the various other functions (in the qa-include/db/ folder) that actually run the queries. From what I recall MSSQL doesn't have a "LIMIT" clause like MySQL.
The search is quite complicated to go into here, but take a look at the functions in qa-include/app/search.php and qa-include/plugins/qa-search-basic.php.
In short: every word from every post is indexed in the qa_words and qa_contentwords tables; when you search those words are looked up and appropriate results returned.
