Pocket underpair with 2 overcards on the flop is the best hand 52% of the time and has a win rate of 75% (when it is best hand on the flop), which means that you can can expect to win 39% of the time overall (.52*.75=.39). Add in the 3 diamonds, and your win rate has dropped even more.

So, I would have folded here too.