Well, there you go then - no problem. It's like entertainment money. If you all had gone out to a bar, played some pool, chased after women and drank it up, he would be out that money anyway - and so would you. This way, at least one of you profits. He's got no reason to be upset with you, so you have to reason to feel bad (except that maybe you could have secured more of his money if you had raised it up on the river).