"I had raised after he limped 3-4 times in the last 10hands before this one, so he is likely annoyed right now"

Wouldn't he be thinking "I'm going to limp with a good hand so I can pop him with aces, AK, AQ ..." rather than "That does it! Im raising with crap just to hurt him!"

btw, I would raise this flop ... but maybe something can be said for letting him bluff his money away & saving money from an extra raise when he does have us beat badly.