I don't think the first hand was an attempt to "pure" bluff you out as he did have somewhat of a hand. It was more of a semi-bluff. He would've "preferred" that you fold, as opposed to "needing" you to fold.

Given that your opponent showed that he is willing to go to showdown with marginal holdings, I don't see how you could fold that second hand. There are very few hands here that beat you. If he had the boat, that's poker.

You simply can't play poker always worried about the worst case scenario, even though you will occasionally run into it.