You didn't really suck out that bad if you filled up. Many people don't realize the improvability of a set. You have 6 cards that pair the board on the turn to fill you up plus the 1 card that gives you quads. This gives you a 14% chance to fill up on the turn. Then on the turn there are 3 more cards to pair the board giving you ten outs to improve or roughly 20% to improve on the river. From your perspective, if he's got the flush then you're a good chance to improve. If he's got a flush draw, then if he misses the turn, then you have more outs than he does. Usually some of your outs that pair the board and such are cancelling out his flush draw outs. It's great when a person makes their flush on the river only to find out that it filled you up. You made a good call and tell him to "suck it" if he calls you a fish when you fill up.