[emanon]

Lets see what everyone thinks of this answer:
Assumptions:
1. Your "hand" is ~ U(0,1)
That is, we are using a univariate distribution to draw each of the hands.
2. All players pursue an optimal strategy.
3. Unlike poker, each hand is independent of the others.

My steps:
1. determine the probability of a victory.
2. determine the EV of a bet.
i. determine the min optimal hand to bet
3. Solve for min victory value.

1. Probability of a victory:
Given you have a hand X, what is the probability that your hand is better than every other players?

P(X>Xi) = Probability that hand X is better than hand of player i.

I'm not sure how to express this compactly, so I will expand P(victory|X) for n=2 thru 4.
For n=2, P(X>X1) = (1-X)
For n=3, P(X>X1 & X>X2) = P(X>X1) + {1-P(X>X1)}*(1-X)
For n=4, P(X>Xi) = P(X>X1 & X2) + {1-P(X>X1&X2)}*(1-X)

Ok, so we now know how to calculate the probability that a hand X will win against n players.

2. Determine the EV.
We want to determine the minimum X that we need to bet.

To determine this, the following variables/equations are needed:
P(min_opt) -> The minimum P(victory) needed to bet.
EV_of_bet -> The expected value for the hands we do bet.
obv -> other bet value, the E[amt of other bets won when i win]
EV -> The overall expected value.

EV = P(min_opt)*EV_of_bet - {1-P(min_opt)}*ante

EV_of_Bet = P(victory)*((a*n)+b+obv) - {1-P(victory)}*(a+b)
Since, when I win, I win all the antes (a*n) plus my bet (b) plus obv; and when I lose, I lose my ante and my bet.

obv = (n-1)* {X - P(min_opt)} * bet
Since: There are n-1 other players who might bet.
Remember, the hands are independent.
For each player, the chance that he will get a hand better than P(min opt) (and therefore bet) but less than my hand (x), is simply the difference between the two.

3. Solve for min victory value:
i. At the inflection point, where EV=0, P(min_opt) == P(victory).

Doing most of the algrebra results in:
0 = (na+2b)*P(min_opt)^2 - P(min_opt)*b - a
Once P(min_opt) has been solved for, need to back out the value X for the min hand to bet.

For a=1, b=2, n=3 I solve for an X=.707

Thoughts, comments, errors, clarifications?
emanon
-------
What about when your opponents are playing suboptimally?
The optimal strategy will no longer maximize your return, however, it will have a higher EV.