With the call on the river, my guess is he's on 2pr, probably KJ, and not the flush draw, and he's just as worried that you're on the flush draw.

Or he could be on some suited connectors that are going to pay off, but in that case he should raise you on the river for value.