Mistaken muck returned by dealer

At the MGM in the NL1/2 game Christmas Day there was like $12 in the pot, 6 limpers, no betting along the way, an ace hits the river, and someone makes a $5 bet. I have a crappy ace but for $5 I'm gonna call, everybody else folds.

The guy who put in the original $5 bet then tosses his cards face down to the dealer. I figure he made a stupid bet and knows it and is surrendering, so I'm putting a $1 chip on my cards and pushing 'em forward when the dealer pushes the cards back to the guy and says "you were called". Guy says "oh!" and flips 'em over showing a better ace than mine.

ok... It was a tiny pot. No big deal. But I quietly asked "why didn't you just push me the pot?" and the dealer said "he obviously hadn't seen you had called." I said no more.

But what's the rule? The guy tossed his cards far past the yellow betting line face down. They hadn't hit the muck. And he didn't ask for them back. Dealer just offered them. If this happens again with $500 in the pot I'm gonna care a lot more.