Well, you're right. He could have a set. But I think it's more likely he's on a super draw. Why would he bet all in? It seems like he is trying to end the hand with the all in. I see this mostly when an aggressive player is on a draw and knows if he gets called, he has a good draw to his monster.

So, yah. You can't totally eliminate a set. But that's my thinking.