If he really is as passive as you say, there is a good chance you are beat. He could have flopped 2 pair or something. You can fold and still have an above average stack and you will probably have a better oppurtunity to get your money in later.