A rotation defense in basketball is a defensive system that rotates players across all five positions in order to have the best defender on the ball. That is different from a standard man to man defense because each defender is not assigned to one specific offensive player.
The rotation defense can also be used in the form of a zone. Players rotate around to different parts of the court rather than to specific players. What that does is allow for the defense to trap the offense, which is explained below.
One defensive strategy teams employ is leaving a member of the offense open. That player is typically left in the opposite corner, not easily accessible by one pass. The point of this is to be able to trap the ball handler. Leaving one offensive player frees up a defender to double team the player with the ball. A diagram depicting this situation is shown below, the defenders (triangles) have three players guarding two offensive players (circles).
Rotation defenses can be very effective slowing down prolific offenses, however they can be dismantled and beaten. The defense is essentially gambling that the offense will not be able to get the ball to the one open player. The problem with this is it isn't actually that hard to do so. Quick ball movement as well as movement off the ball makes it difficult for the defense to keep up and successfully trap.
The number one factor in whether or not a rotation defense will be successful is the effort put in by the defense. The whole system cannot succeed if the defenders are unwilling to exert themselves running to their spot.
Another major key for a rotation defense is awareness and positioning. For it to work, defenders must not overcommit and be able to contain the offense. Similarly, players must be able to help out their teammates if they are beaten by making a quick recovery.
If those two things are done right, then there should be lots of turnovers forced and the offense will struggle to move the ball.