I think a fold is the correct play here. I see no problem with it.

Put your money in there when you are more certain you have the best of it. This is one of those marginal situations where your table image and his come into play.