A hand signal in basketball is used when a player wants to communicate something non verbally to their teammates. They use a hand gesture instead to tell their teammates what the play is for offense or defense.
Isolation is a term used to describe when an offensive player wants to go 1 on 1 against the defense. When a player wants to call for an isolation they will wave their hand as if shooing away their teammates. This motion is also done when a ball handler does not want a pick that their teammate has set.
Hand gestures are an excellent way to simplify calling plays and establish a method for communicating them. A lot of teams assign plays to certain numbers. That way when a point guard wants to run a specific play it is easy for them to communicate what it is to the rest of the team. They just need to hold up the number on their fingers that is assigned to the play.
Some teams call plays with complex hand gestures to give more options than just five fingers of plays. It also makes it harder for the defense to pick up on signals and anticipate what is coming before it does.
A lot of players also use hand signals while celebrating during the game. A common celebration is the extending of the pinky finger, ring finger, and middle finger while connecting the index finger and thumb to celebrate a three pointer.
Custom handshakes are also popular. Players like to have unique handshakes or "daps" with each teammate.