[BruceZ]

$1.78 is what he should pay if he gets his money back when he wins, in addition to the money he wins.

EV = 8/52 * 5 + 8/52 * 3 - 36/52 * x = 0

x = 1.78

If he doesn't get his money back, then he should only pay $1.23.

EV = 8/52 *(5-x) + 8/52 * (3-x) - 36/52 * x = 0

x = 1.23