Testing ICM -- some questions for discussion
 

I've been rereading some threads on testing ICM, including:

eastbay's empirical equity study,

eastbay's post in ICM: Does it match the results , and

schwza's computer programmers (x-post) .

For those who would like some background on the subject, check out Section 4.1 of my favorite threads list.

It seems to me that the simplest empirical test of a chip modeling hypothesis is whether the equity function -- the relationship between chip value (CEV) and equity ($EV) -- is linear in a heads-up freezeout. Sklansky seems to assert that the relationship is linear (TPFAP p. 151) ("[E]qual players in a symetrical situation must win exactly in proportion to the size of their stacks."). If true, the probability of hero winning equals the hero's chips as a percentage of total chips. If the hero holds 75% of the total chips, the hero's probability of winning is 75%, etc., etc. As I understand, ICM similarly assumes linearity, at least in freezeout format.

eastbay, zephyr, and others have questioned that hypothesis. JNash has proposed a S-Curve Hypothesis . He conjectures that the equity function is convex for small stacks and concave for big stacks, with the inflection point equaling average stack size. JNash speculates that the S-curve becomes more pronounced as blinds increase relative to the average stack. If true, the probability of hero winning would not necessarily equal the hero's chips as a percentage of the chips in play.

As far as I know, no one has tested whether the equity function is linear in a HU freezeout. While many other aspects of chip modeling are both interesting and await empirical analysis, such as incorporating a skill factor, this aspect strikes me as more tractable and interesting. Focusing on 2-person play would avoid many of the computational and data collection problems involved in 3- or more person play.

Even so, the result would still be interesting. As JNash suggests, under the S-curve hypothesis, it would be correct (+$EV) for the small stack to take a perfect coin flip (50/50 CEV), but incorrect (-$EV) for the big stack to accept the same coin flip. To put it differently, it could also be correct (+$EV) for the small stack to take some -CEV gambles.

So, this reasoning leads me to some questions for discussion. First, can anyone else suggest a better (i.e, simpler and still interesting) aspect of chip modeling to test?

Second, if we were to test this aspect empirically, what data should be collected? It seems to me that it would include the following fields:
Data source
Site (Party, Stars, etc.)
Buy-in ($5, $10, $20, $50, $100, etc.)
Tournament no.
Chips for player A
Chips for player B
Winner (1 for A, 0 for B, or some similar convention)
Are there any other fields that should be collected?

Third, what should be the size of the data set? Should n=500, 500,000, 5 million, etc.?

Fourth, related to the second question, what tests should be run to test the data for linearity or non-linearity? I recall several such tests from my econ days, but recall that they dealt with time series. I am not sure of their application to this problem.

Finally, aside from collecting data, are there other ways of testing the hypothesis of a linear equity function in a HU freezeout?

I have some thoughts of my own, but thought it might be best to pose some questions for discussion. I welcome any suggestions and comments.

The Shadow

P.S. If you find it hard to visualize JNash's S-curve hypothesis, take a look at M.C. Escher's print Concave and Convex. A simple S-curve suddenly seem a lot easier. [img]/images/graemlins/smile.gif[/img]

EDIT: Links edited out to fix legibility problem.

Re: Testing ICM -- some questions for discussion
 

Please edit for readability.

Re: Testing ICM -- some questions for discussion
 

It was fine when I previewed it. I'm editing it now.

Re: Testing ICM -- some questions for discussion
 

OMG fix it. Not that I would probably understand it anyway.

Re: Testing ICM -- some questions for discussion
 

Shadow,

I think it's a good place to start. The difficulty I possibly see is that any deviations from linearity may be small, and since what we'll be looking for is small deltas, it will again be hard to find them (with any confidence) without huge sample sizes.

BUT, I think it's worth trying anyway, and if you do find the S-curve phenomenon, fame and riches await you. Well, not really, but it would be a very cool result nonetheless.

Before we get into questions of sufficient sample sizes, etc, do you have something in mind for how to do the data mining?

eastbay

Re: Testing ICM -- some questions for discussion
 

I have a few ideas.

First, the best place to start is with data from actual tournaments. The best place to get that data is from online sites. If the data to be collected were clearly defined, one might be willing to cooperate. Poker Room, for example, has posted the limit EV data that you and I have debated before. Since the data that would be collected would not reveal any site-specific or player-specific information, one or more sites might just be willing to help out.

Second, even if no online site were willing to assist, it might be possible to collect the data from a group of players, especially if the data set required for any meaningful analysis would not be insurmountable.

Third, if online sites were not cooperating and sufficient data were not available from players, I would give some thought to taking data from bot v. bot simulations. There are a number of increasingly sophisticated poker bots available on the market. While I would have concerns about the validity of applying bot data to human play, it might be illuminating in this context. For example, if the bot data were consistent with a null hypothesis of a linear equity function, my faith in that assumption would be different than if the data suggested a non-linear relationship.

Fourth, aside from collecting tourney or bot data, there are other ways to attack the problem. For example, online electronic markets have been used to predict presidential elections and economic indicators. One way of approaching the equity function would be to ask, how much money would a third-party be willing to pay for a stack size of x in a HU freezeout? There are ways, such as online markets, to determine that answer without having to collect and analyze a gazillion data points.

The Shadow (who appreciates your patience while I fixed the links)

Re: Testing ICM -- some questions for discussion
 

If you want an "empirical" study then, right, you have to collect the data, and online games is the only viable option. I am doubtful that you could get cooperation from the sites.

I think the best way to do it is with an observed game data mining program. This is a significant undertaking, but has obvious multiple utilities which may justify the "cost" of developing such a capability.

Collecting data from individual players has clear sample bias problems if you are looking for effects which occur in the mean of the pool of all players.

An alternative is with a game model and simulation. I have done this with some quite simple "push/fold" strategies. The results are remarkably linear, but the strategies may not include "real world" elements that generate the deviations from linearity. I have some pretty well-defined ideas about how to improve such simulations to search for the S-curve phenomenon. But any such study is not really "empirical" and there will always be doubts about the sufficiency of the strategy algorithms to capture all of the things that real players do which might generate the deviations from linearity.

Just a quick brain dump.

eastbay

Re: Testing ICM -- some questions for discussion
 

Thx for your thoughts.

The data mining idea sounds promising, but would be beyond my skills.

As a lawyer, I'm continually surprised by how much information you can get by simply asking. Assume that an online site were willing to assist. What and how much information would you be looking for?

Re: Testing ICM -- some questions for discussion
 

I actually think the bot vs. bot idea is a very good one, since having the same bot play each other would completely remove any bias in favor of the better player. This would give an excellent control set for any chip ratio you might desire.

I'm not sure how fast a computer could run a bot v bot heads up match, but in theory you could generate a substantial amount of data relatively fast with this method.

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
As a lawyer, I'm continually surprised by how much information you can get by simply asking. Assume that an online site were willing to assist. What and how much information would you be looking for?

[/ QUOTE ]

The size of the blinds at the time the match gets to be HU seems like it would be a relevant factor.

One issue that interests me about the data collection is how many data points one should take from each tournament. Let's say I play 20 hands heads up against my opponent; should we use this as 20 data points? My gut feeling is no, we shouldn't, because any deviations from average behavior become magnified.

As an oversimplified, unrealistic example of what I'm talking about, say I get heads up with blinds of 100, my stack being 2000, villain's stack being 8000. Now let's say he's ridiculously tight, so tight, in fact, that he'll fold anything but aces. Now I should win this just about all of the time, and so I'm going to skew the data horrendously if you take multiple data points from this tournament that show my small stack overcoming a major disadvantage and winning.

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
[ QUOTE ]
As a lawyer, I'm continually surprised by how much information you can get by simply asking. Assume that an online site were willing to assist. What and how much information would you be looking for?

[/ QUOTE ]

The size of the blinds at the time the match gets to be HU seems like it would be a relevant factor.

One issue that interests me about the data collection is how many data points one should take from each tournament. Let's say I play 20 hands heads up against my opponent; should we use this as 20 data points? My gut feeling is no, we shouldn't, because any deviations from average behavior become magnified.

As an oversimplified, unrealistic example of what I'm talking about, say I get heads up with blinds of 100, my stack being 2000, villain's stack being 8000. Now let's say he's ridiculously tight, so tight, in fact, that he'll fold anything but aces. Now I should win this just about all of the time, and so I'm going to skew the data horrendously if you take multiple data points from this tournament that show my small stack overcoming a major disadvantage and winning.

[/ QUOTE ]

Intuitively I agree that only one data point is permissible on independence of observations grounds. I'm not sure how to prove it, but I'm not sure that it needs proving.

eastbay

Re: Testing ICM -- some questions for discussion
 

If the linearity theory is indeed false, it would not be hard to disprove. You only need to do an analytic or simulated solution for three data points under one set of conditions. If you find a non-linearity in those (which I'm sure can be done given a correct choice of data points), then the equity function must be non-linear. In other words, you basically only need one counterexample.

On the other hand, I would think "proving" the s-curve hypothesis would be quite difficult.

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
If the linearity theory is indeed false, it would not be hard to disprove. You only need to do an analytic or simulated solution for three data points under one set of conditions. If you find a non-linearity in those (which I'm sure can be done given a correct choice of data points), then the equity function must be non-linear. In other words, you basically only need one counterexample.

[/ QUOTE ]

I'm not clear on what you mean by this. The whole point of the exercise, as I understand it, is to see how well the analytical models match up against reality. Whether or not some specific model predicts linearity isn't really the question, but it seems to me that you're suggesting doing calculations to determine just that.

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
The whole point of the exercise, as I understand it, is to see how well the analytical models match up against reality

[/ QUOTE ]

To me, data collection simply tests the assumption of symmetric players. If you assume symmetric players, the problem can be solved analytically (or approximated via simulation).

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]

To me, data collection simply tests the assumption of symmetric players. If you assume symmetric players, the problem can be solved analytically (or approximated via simulation).

[/ QUOTE ]

In what model? I still think you're missing the point.

Re: Testing ICM -- some questions for discussion
 

Seconded.

Don't confuse models with reality. The whole point of making measurements is to validate a model, whether it be a relatively simple ICM or a more involved full-scale simulation.

eastbay

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
[ QUOTE ]

One issue that interests me about the data collection is how many data points one should take from each tournament. Let's say I play 20 hands heads up against my opponent; should we use this as 20 data points? My gut feeling is no, we shouldn't, because any deviations from average behavior become magnified.

[/ QUOTE ]

Intuitively I agree that only one data point is permissible on independence of observations grounds. I'm not sure how to prove it, but I'm not sure that it needs proving.

eastbay

[/ QUOTE ]

I don't see how this would create a bias in the data. You would be doing this for every tournament so the fact that you have 20 points of data from one tournament where one player was exceedingly tight is countered by the fact that you have hundreds of points of data from just a few dozen tournaments where players were reasonably loose.

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]

I don't see how this would create a bias in the data. You would be doing this for every tournament so the fact that you have 20 points of data from one tournament where one player was exceedingly tight is countered by the fact that you have hundreds of points of data from just a few dozen tournaments where players were reasonably loose.

[/ QUOTE ]

But consider that in loose contests, it's likely to end faster, so you will have proportionally fewer data points for the same number of SNGs. I think the simplest thing is just to take one data point per tournament.

Re: Testing ICM -- some questions for discussion
 

We agree to disagree.

Re: Testing ICM -- some questions for discussion
 

Can you please explain in the most simple English possible what precisely it is that you're going to do, perhaps by laying out in detail what calculations you think you can perform and what they'll demonstrate?

Re: Testing ICM -- some questions for discussion
 

Sure. My understanding is that the issue in question (the linearity of the equity function) is basically the same as saying that someone with x% of chips in a HU freezeout with someone of equal skill will win x% of the time. My point is that if that premise is false, to disprove it does not require extensive data collection. All you have to do is to construct a scenario where someone with x% of the chips wins something other than x%.

That says nothing about the s-curve or any other possible shape for the equity function. But if it can be done, it shows that the function is not linear in all cases.

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
We agree to disagree.

[/ QUOTE ]

What I said is not a matter of opinion.

eastbay

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
What I said is not a matter of opinion.

[/ QUOTE ]

No, but some of it is incorrect, specifically the part about the whole point of making measurements being to validate a model.

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
. All you have to do is to construct a scenario where someone with x% of the chips wins something other than x%.

[/ QUOTE ]

This is a very vague statement. We can construct scenarios where opponents have equal chips but one always loses, because the other one has AA. This isn't what you mean, but you haven't nailed down what you mean. What eastbay and I are trying to get at is that you can't nail down what you mean in any way that will answer the original question. Any calculation or simulation you make will have to assume certain strategies, certain values of the blinds, etc., and now you're not really talking about heads-up poker, you're talking about a model of heads-up poker.

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
[ QUOTE ]
What I said is not a matter of opinion.

[/ QUOTE ]

No, but some of it is incorrect, specifically the part about the whole point of making measurements being to validate a model.

[/ QUOTE ]

Uh... whatever.

Do you know what ICM is? Do you understand what validating a model means?

eastbay

Re: Testing ICM -- some questions for discussion
 

OK. I see what you are saying. My understanding was that Sklansky's assertion was basically that over a large number of trials those things really didn't matter, so long as the players were of equal skill. If this is not what he is saying, then you are correct -- what I said can't be done.

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
Do you understand what validating a model means?


[/ QUOTE ]

Yes. I have been doing academic research for the last 9 years, developing and validating models and publishing the results. There are lots of reasons to take measurements and lots of ways to validate models. They are not one and the same. Like I said, we agree to disagree.

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
[ QUOTE ]
Do you understand what validating a model means?


[/ QUOTE ]

Yes. I have been doing academic research for the last 9 years, developing and validating models and publishing the results. There are lots of reasons to take measurements and lots of ways to validate models. They are not one and the same. Like I said, we agree to disagree.

[/ QUOTE ]

By definition there is only one way to validate a model: make measurements.

You may be confusing validation with verification or you may use the word in some kind of sloppy way which considers model to model comparison "validation." I don't, I expect anyone who is involved in academic research to know better.

eastbay

Re: Testing ICM -- some questions for discussion
 

For lack of a better thread to put it (this is enough of a triviality that I don't feel it merits its own complete post), here's an argument regarding heads-up linearity. It's not a very complicated argument at all, but it at least backs up the intuition that % stack should equal % win in a particular case. This is the kind of thing that seems very intuitive to me, but I'd never looked at any kind of argument for why it had to be.

The case that I'm considering is simple move-in poker - two players, with stacks x and y (assume without loss of generality that x > y), both go all-in every turn. Since they are pitting two random hands against each other, each has a 50% chance of winning each confrontation. We can now easily show that the equity formula E(x) = x / (x + y) is consistent. Half of the time stack x wins, half of the time stack x ends up in a new game with stacks x - y, 2y. So,

E(x) = .5 + .5 E(x - y) = . 5 (x + y / x + y) + .5 (x - y / x + y) = x / (x + y)

Notice that if we choose some kind of screwy function, like E(x) = 1 if x > y, 0 otherwise, for example, we don't necessarily get this consistency. So it's a strong argument that % stack = % win works for this particular situation. Also, it's important to note that this just assumes a relatively braindead strategy of pushing and calling every time; an actual game is more nuanced, which is where the argument comes in.

Re: Testing ICM -- some questions for discussion
 

This thread is in full hijack mode largely because of me, so this will be my last post on the subject.

Yes, I am all too familiar with verification and validation. I understand everything you have said, and I disagree with the points I noted. You think I am completely wrong, and I am fine with that. Life would be boring if we all agreed on everything. All the best. [img]/images/graemlins/smile.gif[/img]

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
This thread is in full hijack mode largely because of me, so this will be my last post on the subject.

Yes, I am all too familiar with verification and validation. I understand everything you have said, and I disagree with the points I noted. You think I am completely wrong, and I am fine with that. Life would be boring if we all agreed on everything. All the best. [img]/images/graemlins/smile.gif[/img]

[/ QUOTE ]

Then I have no idea what you could disagree with but I don't care enough to find out either.

eastbay

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
All you have to do is to construct a scenario where someone with x% of the chips wins something other than x%.

[/ QUOTE ]

Hi,

This sentence is one of those things that *sounds* clear on its face, but is actually pretty murky. I'm going to rephrase some of what you said in words that I understand .. can you tell me if this is what you mean?

The claim: Given two opponents with identical strategies, chip value is linear in a heads-up freezout.

Oops .. I'm going to interrupt here .. we have to mean something specific by "identical strategies" .. if our strategy is to wait for a huge hand and fold a lot with a small stack, but push a lot with a big stack, we're dead in the water, right? Ok, let's assume we can ignore that .. does making the opponents play a good, pure, jam-or-fold strategy work? Intiutively, I think it might. Jamming/calling every hand has to be linear, but it seems a bit too trivial.

Right, back to the question at hand. If chip value is linear, we can put things on a graph where the X axis is the number of starting chips and the Y axis is the percentage of tournaments won if two opponents sit down and play all possible tournaments at this structure (i.e. every possible run of cards).

Two points make a line, so if we can find three non-colinear points, we know the claim is false. Heck, we don't necessarily need to do more than one calculation: we already know that (0,0) and (100,100) are on the line.

So, what you want to do is this:

1) pick a strategy that isn't obviously broken (e.g. the pure jam-or-fold strategy I mentioned earlier).

2) pick a structure that you think might lie off the 45-degree line (a very small stack is probably a good bet, but it must be larger than the blind).

3) run a *ton* of simulations looking for a monte-carlo-type conclusion about y(x).

if you get a value that's off of the 45-degree line in some statistically significant way, and you can convince yourself that your strategy isn't broken, you win.

right? is that what you meant? i'm tired, so i could easily be misinterpreting.

Re: Testing ICM -- some questions for discussion
 

Basically yes, although I think you could just see if the actual percentage of wins matched the expected percentage under linearity instead of plotting.

Re: Testing ICM -- some questions for discussion
 

1. To poincaraux:

[ QUOTE ]
(a very small stack is probably a good bet, but it must be larger than the blind)

[/ QUOTE ]

Why do you impose the requirement that the small stack must be greater than the big blind in your hypothetical simulation? I have some thoughts myself that suggest why the equity function is non-linear once the small stack is less than the big blind, but would like to hear your reasoning.

2. To everyone:

Assume that we collect a gazillion data points from actual SNGs. Assume that each one is from a separate SNG to avoid any arguments about the independence of the data. Assume that we get the following information:

We have n number of observations where player A's chip stack (Ca) is x fraction of the total chips in play (Ct). At that stack size, player A wins a mean of y fraction of the time with a standard deviation of s.

Assume that x > y. In other words, on average, player A apparently wins less than than what a linear equity formula would suggest.

How large does n need to be for us to say with confidence that y lies off of a linear equity function?

Of course, we have data for not only x=1, but also x=2, x=3, x=4, . . . where 0 < x < 1. What should we do with that data?

The Shadow

Re: Testing ICM -- some questions for discussion
 

hey all,

just noticed this thread.

here's an update on my progress (see link in original post)that touches on some things in this thread. i found a shareware program that i'm using to automate datamining. it does 4/hour now, but i can increase it by adding more than one skin and reducing the time between opening and closing stt's.

a 2+2'er has agreed to write a text parser to turn hand histories into data. and i'm in touch with an econ professor at u. md. who is going to work on the data analysis. he said that it is possible to use multiple data points from the same stt using some clever statistical technique. he also said that 250 stt's would be enough to start looking at preliminary models.

we're going to look at heads up first. i'll keep you posted.

Re: Testing ICM -- some questions for discussion
 

Assume that a drunk is standing 1' to the right of a cliff. It's dark, so he doesn't realize it, but he's walking parallel to a cliff, which runs in an absolutely straight line. If he takes a single step to the left, he falls to his death.

For every step that the drunk takes forward, he lurches 1' either to his left or right. As best as we can tell, he veers left or right absolutely randomly.

The drunk takes 1 step forward every few seconds or so. He never sits down, stands still, or moves backward.

Luckily for him, 10' to the right of the cliff is a shallow ditch filled with soft clover. If the drunk reaches it, he falls down and gets to sleep it off. (When he wakes up, he can start multitabling SNGs all over again. [img]/images/graemlins/grin.gif[/img])

To put it visually, the drunk is walking north towards the top of the screen:

Cliff Safety
C--------S
C--------S
C--------S
C--------S
C--------S
C--------S
CD-------S

What are the chances that the drunk will fall to his death?

What are the chances that the drunk will wake up in a bed of clover?

What does this have to do with whether the equity function for a freezeout SNG is linear or not? (I'm not sure but have some thoughts.)

The Shadow (who is glad that no drunks were actually harmed in this hypothetical)

Re: Testing ICM -- some questions for discussion
 

I'm not sure I see how this random walk problem meshes well with HU poker. Perhaps you are suggesting that each step to the right is a double up, and that our intrepid hero will need to double up 10 times to win the tournament?

Let D(x) be the probability of dying starting at 1', 2', 3', . . . 9'. If he can make it to 10', he lives. We can come up with a system of equations for determining the equity of each step:

D(1) = .5 + .5*D(2)
D(2) = .5*D(1) + .5*D(3)
.
.
.
D(9) = .5*D(8)

It looks to me, spending a minute working on this, that D_1 = .9; that is, starting where he does, he makes it to the clover 10% of the time. If we go with the assumption that you intend steps to be doublings, then he wins 10% of the time although he's going to have something on the order of 1 / 2^9 of all chips, so there's a nonlinearity here.

Interesting. I'll have to think about this some more.

EDIT: Okay, here's the flaw in this argument: until such point as the hero has acquired half of the chips, he's always effectively 1' away from the cliffside, so this isn't really an analogous situation; after doubling up, he's either ahead, in which case he's 1' away from the clover, or behind, in which case he's 1' away from death. The intermediate steps can't exist.

Re: Testing ICM -- some questions for discussion
 

I didn't read carefully but this sounds like a random walk. A random walk in the continuous limit is governed by a diffusion equation. Diffusion analysis of tournament equity has been studied quite a bit.

The solution to the diffusion equation is indeed exactly linear in 1D (HU game).

Here's an exact solution for the 3-player problem:

http://www.math.ucla.edu/~tom/papers...mblersruin.pdf

Bozeman compared ICM to the diffusion solution in a great post quite awhile back. They are surprisingly close. You'll have to search for it if you're interested.

eastbay

Re: Testing ICM -- some questions for discussion
 

[ QUOTE ]
hey all,

just noticed this thread.

here's an update on my progress (see link in original post)that touches on some things in this thread. i found a shareware program that i'm using to automate datamining.

[/ QUOTE ]

Care to share which one? Keep us posted.

eastbay

Re: Testing ICM -- some questions for discussion
 

schwza,

Thx for the update.

It occurs to me that when you gather and analyze the data, you need to record seat position in addition to stack sizes. It would not be enough to simply call the short stack "Player A" and the big stack "Player B." If you did that, I would expect to find that Pa (probability of A winning) < Ca/Ct (A's chips/total chips), especially towards the extremes of the range (A down to 25% or less of the chips to pick an arbitrary number). After all, someone has to win the freezeout. If the chips have flowed from A to B, maybe that reflects a skill difference between the two players. If so, maybe B can do more with the big stack than a linear equity function would suggest.

To avoid the problem, assign "Player A" to the lower seat number and "Player B" to the higher seat number. That way, we should still be able to preserve the assumption of equal skills on average between the two positions.

The Shadow