Short-Handed Blind Question

This situation comes up alot and I'm not quite sure how to play it. Button raises, SB calls, I have J9o in BB. If it's HU against the button I call, but if it's 3-way I fold it figuring the button could have anything but the SB has a slightly better than average hand. SB combined with the button's possible holdings makes me think that I don't have a good EV on this hand. Is this being too timid? I don't know if folding is a mistake or not, I figure it's one of those close calls but I'm not sure. J9o against 2 random hands will win 36.5% but since I figure the SB doesn't have a "random" hand the win % would probably be considerably lower. If there was going to be no more betting and we were just going to showdown I'm getting 5-1 odds so I guess it would be worth a call. But if I had to call any more bets I don't think the odds would be there to play that hand.

I know this decision is somewhat opponent dependent but assume you just sat down and have no information on your opponents.