I think you have to push mainly because if you hit your hand it's pretty much nothing but scare cards for him. Only a set with some of your outs type hand is a fav over what you have (and even then it's not that big)

Plus when you push then they might start calling those with hands like middle set when you have top set and etc.