I think your line is OK, but I don't think many river cards will hurt you since you're fairly sure he's not on the flush draw.

I'd get another bet in on the turn, when he will call. Then the river pot will be so big he'll call another bet in case you were bluffing.

Oh, and call any reraises and call down from there.