Tourney or cash game? If cash game, how much did he have? If a guy has a full buy-in, chances are, unless he thinks you're stealing, he's not going to put it all in without aces or kings. Fold.