A 'trey' in basketball is another name for a three-point shot attempt. The term is widely used by announcers and color commentators to describe the action during a live TV broadcast. Other synonymous names include 'trey ball' and 'downtown.'
A three-pointer is a shot attempt that takes place from behind the three point-line, which is an arch that spans the entire width of the court on each half. Since the three-point line is located somewhat far away from the basket, successful makes are worth three points as opposed to two points, which would be awarded for a shot attempted closer to the hoop.
'Trey' is used merely to describe a shot attempt from behind the three-point. Whether or not the shot is successfully converted generally has no bearing on the usage of the term. However, announcers will often say something along the lines of "hits the trey ball" or "can't convert the trey" depending on whether the attempt was made or missed.