Viability in fantasy football means the same thing as it would in any other context. If a player is viable, that means they are capable of producing enough fantasy points to return the cost of investment.
One might say, "it isn't viable to draft that player in the first round." This means that the player wouldn't return the cost of the first round selection. This could be due to many things, but it's most often player performance, injury risk, or the player's team situation.
A player's viability is primarily reliant on their performance. Performance is the main thing that makes a draft selection, a free agent pickup, or a trade viable. If a player performs well, then they are viable, and if they perform poorly, they aren't viable.
Injury risk can also affect a player's viability. If a player is "injury prone" then drafting or trading for them is likely to be a lot less viable.
Something that flies under the radar a lot when considering a player is team situation. For example, a wide receiver on a team with a subpar quarterback might be a lot less viable than a different wide receiver with a great quarterback, even though the two wide receivers might have similar skill levels.