Mental Models are one of the most important concepts when it comes to the user experience. The term has been coined (as far as I know) by Don Norman, and his colleague Jakob Nielsen explains them in this post.
A mental model is an idea of how a product works internally. Problems arise when the user has a different model in mind than the one actually implemented. Possibly the users expects another result than he receives, or he does not understand how to use the product in the first place.
Jakob Nielsen suggest two options is mental models differ from each other:
- Make the system conform
- Improve users’ mental models
It is impossible to summarise all the details, but I highly recommend the article by Jakob Nielsen, which explains it nicely and provides some examples.