Calling here is not a bad play because it is such a big call opposed to the previous acts by villian. The problem is that if he had the flush or a higher straight he would have bet more on the flop and turn. The $20 bet was obviously an overbet of the pot on his part and you should have reraised his bet for a sure fold.