Hope I have understood you question properly!
1. Go to admin, and then to emails section. Disable the "Provide a form for users to send me feedback" option, so that the feed back form is not shown.
2. Go to admin and then to pages. Click on "Add Page". Enter the "Name of page (also used for tab or link):" as Feedback or Contact. In the text area (Content to display in page (HTML allowed): ) Enter your address, phone number, etc, and again click on Add Page.
You are done. The default feedback form has been disabled, and when somebody clicks on Feedback / Contact (page), he / she will see your contact address.
Hope this is what you wanted.