You didn't post what the river was, but I'm assuming that it didn't complete the straight draw. You also didn't say it the aggro player was the original raiser or not. That is important here.

I definitely would have raised in this position with top 2 pair, there are only three realistic hands (2-2, 4-4, 10-10) that you don't want to see. His line doesn't look like a set at all. If he was the original raiser he could have 10-10 and turned a set, but he probably would have raised more with 10-10 or Q-Q to protect his hand.

On a metagame level, I hope you used that tight image to steal later on.