A mismatch in basketball is when a weaker player has to play against a superior player. This can create holes offensively or defensively as you can lose a scorer due to his inability to play better than his defender or vice versa. It can also refer to a big man posting up a smaller defender or a smaller offensive player being guarded by a big man at the three-point line.
Some factors that can create a mismatch include size, skill, basketball IQ, and intelligence. If a player is significantly smaller than another player this can create problems as they can be shut down or outrebounded. Players who are smarter are able to see opportunities that others don't and if there is a large gap in intelligence between players this can create a significant mismatch either offensively or defensively.
There are multiple ways to handle a mismatch. One way to do so is by helping the player who is mismatched with a double team if he is on defense or by getting open and taking advantage of scoring opportunities when on offense. Another way is to not help that player at all, and instead to combat this hole by exploiting a mismatch of the opponent and taking advantage of it.