I'd find it very tough to walk away without saying anything, and yet I know that is the best way to upset this guy, so this is a tough call and I wouldn't KNOW how I would react til it happened (hopefully never does).
Best thing to do though is keep it simple I think. Either walk away or just tell this dude to take a flying leap and then walk away.