a thought on uncertainty

[Slim Pickens]

Just a quick one before I head off for the weekend...

There are two kinds of uncertainty: intrinsic and extrinsic. Intrinsic uncertainty in a value arises from the essential nature of the process that generates that value. It is unavoidable, no matter how perfectly an experiment is conducted. Extrinsic uncertainty stems from either systematic or random changes in the conditions or process by which the value is generated. A perfectly-conducted experiment has no extrinsic uncertainty.

In SnG terms, if a bot played 10,000 sets of 100 tournaments (for a total of one million tournaments), we would expect the results to be distributed somehow, rather than all falling at the same value. No matter how sophisicated you make the bot, adding in randomness and all, it's still a mechanistic device. Every play happens because of some predetermined set of conditions, all of which can be defined. Even though the bot plays this way, the results of any two sets of 100 tournaments can be different because the mechanics of a hand of poker has a strong component of randomness, and that creates the intrinsic uncertainty. Now make a real human play 10,000 sets of 100 tournaments. The human's results would also be distributed somehow, but we could very well expect the distribution to be (statistically) significantly different from the bot's distribution. Why? Components of extrinsic uncertainty. The human does not play entirely deterministically/probabilitically, meaning that the human's play can't be defined the way we were able to for the bot's play.

Extrinsic uncertainty is extremely difficult to quantify. The question of "Is my ROI of XX.xxxx% over YYYY tournaments good?" can only be answered as well as our ability to control and quantify the extrinsic uncertainty. Are my experimental conditions changing as I play? Extrinsic factors such as time of day/week/month, number of players on the site, and so on, may be quantifiable, but their effects might not be, and this adds to extrinsic uncertainty. Even the bot isn't immune to these. The human also has random nerve firings, gut feelings, and distractions, and learns things, all of which influence the results by adding to the extrinsic uncertainty.

My point is this. We can quantify the intrinsic uncertainty in a sample set of tournaments. Anyone who doesn't agree with this doesn't understand a fundamental aspect of poker: a lot of it is random, but probabilistic. What we can't quantify is all the extrinsic uncertainty. IMO we collectively could, but it would be a massive waste of effort. The best question to ask is "When does the intrinsic uncertainty in a sample set of SnG's get small enough so that the extrinsic uncertainty drowns it out?" As a guess, I'm going to suggest 500 for ITM% and 2000 for ROI%. Those are based mostly on the little modeling programs I wrote during one of those statistical significance arguments, but it's not the result of any quantitative statistical arguments. I don't care exaclty where that line is drawn based on desired confidence intervals, but it can be drawn based on solid statistical arguments. I'm giving up trying to refine numbers on my results once they get within what I think is the extrinsic uncertainty. Anyone care to venture a guess as to how big that uncertainty is?

Slim

[Bigwig]

I can only read the words 'intrinsic' and 'extrinsic' so many times in a 2 minute period. I'm sure what you said is very helpful and accurate. Nevertheless...

*head explodes*

[Tilt]

I guess, said another way, how do we quantify how much different people are than bots?

[Pokerscott]

[ QUOTE ]
The human does not play entirely deterministically/probabilitically, meaning that the human's play can't be defined the way we were able to for the bot's play.


[/ QUOTE ]

That doesn't make any sense to me. For any given decision at the poker table, there are a finite set of responses (fold, call, raise x, x+1,...,all-in - 1, all-in).

Every human has a probability distribution for selecting from this finite set of choices. It is true that the human may have a distribution that is conditional on their mood, but that just means you want to integrate over moods before claiming you have the right distribution.

If you buy that the human has a probability distribution over the range of choices, why again is that different from a bot with the same distribution?

Pokerscott

[Gramps]

Off topic, but for some reason, reading that post made me picture Einstein going into a fit of rage and breaking his monitor after getting sucked out on on the bubble for the 4th time in a row by a dominated hand.

Somehow, I think Heisenberg would have rolled with the swings a little better.

I've read Physics For Dummies.

Go Bears.

Oh yeah, I've had 500 SNG groupings where my ITM% differed by over 5% - guess my "extrinsic variables" were really alive and kicking. Though I would guess (from experience and PT) that it's +/- 1% or less most of the time (intrinsic wise).

[Slim Pickens]

[ QUOTE ]
[ QUOTE ]
The human does not play entirely deterministically/probabilitically, meaning that the human's play can't be defined the way we were able to for the bot's play.


[/ QUOTE ]

That doesn't make any sense to me. For any given decision at the poker table, there are a finite set of responses (fold, call, raise x, x+1,...,all-in - 1, all-in).

Every human has a probability distribution for selecting from this finite set of choices. It is true that the human may have a distribution that is conditional on their mood, but that just means you want to integrate over moods before claiming you have the right distribution.

If you buy that the human has a probability distribution over the range of choices, why again is that different from a bot with the same distribution?

Pokerscott

[/ QUOTE ]

I think it's impossible to quantify the probability distribution for the finite set of human actions in a given situation because that distribution is based on an infinite set of inflencing factors. For any one factor, like mood, you can integate over all of them and get a mood-independent distribution, mean value, and variance. The problem is that there are an infinite (well, effectively infinite anyway... I don't want to start a "nature of the universe" thing here) number of factors influencing play, and that's why you can't get a definite number for extrinsic uncertainty. The same logic applied to the bot also says there's an infinite number of external factors (time of day/week, players on, etc.), but the human has the infinite number of "human factors" as well, making his uncertainty different than the bot's, so we can't just set a bot loose on Party Poker and nail down the total extrinsic uncertainty.

Slim

[Pokerscott]

I think you are getting caught up in the complexity of the situation. To simplify, imagine someone tosses a coin and you are asked to 'call it in the air'. You have exactly two choices, heads or tails. If I am hearing you right, you are saying there is no way to create a bot that mimics a human because the we can't define the probability that the human says 'heads' or 'tails'. Sure, based on the randomly impacted whim of the human, the answer may be either, but that doesn't matter. All you need to know is what is the unconditional probability of picking one of the finite choices.

Pokerscott

[skipperbob]

I wish you would have been my "statistical analysis" teacher; maybe I woulda "got" it /images/graemlins/confused.gif

[Slim Pickens]

Yeah, I might be making this too complicated, but I stand by my point. We're only telling half the story when computing statistical confidence intervals for SnG results because those don't necessarily account for all the external factors. While this doesn't make that sort of analysis totally invalid, since it should account for all intrinsic uncertainty and a portion of the extrinsic uncertainty, I think putting absolute uncertainties on real people's observed mean values by assuming all extrinsic factors are random and normally distributed is wrong. There is an absolute limit on the degree of certainty with which one can say "My ROI is 41.721% +-0.xxx% because I have played one jillion tournaments," and that limit is set by the extrinsic uncertainty. Therefore, people shouldn't get excited about playing large numbers of tournaments to shrink their statistically calculated confidence interval.

Slim

[The Yugoslavian]

Aleo used to wonder about this very problem (if I understand you correctly, which I very well may not) all the time...

You should like, pm him and stuff....

I've decided (after reading about and experience all sorts of variance and 'statistics') is that the stats themselves really are meaningless (in any rich way)....they are very useful for discussion though.

Yugoslav

[Irieguy]

Nice post. Somebody finally put into words what's REALLY the compelling reason to question the validity of statisical analysis of SNG results. Formerly, the best argument was "it's not a normal distribution," and this turned out not to be correct emperically.

I'll have to think about how big extrinsic uncertainty rates to be. Your post implies that it's pretty big, whereas the common assumption with regard to statisitical analysis (of SNGs) has been that it's quite small. I'm interested to hear if you have some reason to think that this is knowable... particularly if you know it.

Irieguy

[Slim Pickens]

I don't claim to know the value or even the magnitude of extrinsic uncertainty. There are three experiments I think could tell us something with a large enough sample size.

1) A bot playing against a field of bots will tell us the intrinsic uncertainty, or you could just calculate it from statistics.

2) A bot playing against a field of human players could quantify the non-human extrinsic factors by comparison to the bot-vs-bots results.

3) Finally, you could take all the people-vs-people data (say from all 2+2 STT posters who keep stats against the actual human field) and compare this to the bot-vs-people results to quantify the extrinsic human factors.

Slim

[Slim Pickens]

I thought about it a little and I'm going to take a guess as to the value of the extrinsic uncertainty in SNG tournaments. It's about 3% in a typical player's long-term ROI, and most of that is from skill drift relative to the field.

All confusion about this subject could be removed simply by stating: "The uncertainty in a player's ROI as calculated by mathematical formula, such as in the AM spreadsheet, is no less than the value given." It can be more, sure, but it's absolutely not less.

Slim

[pokerlaw]

I like this discussion. Some thoughts i had...

The reason that I agree only a bot can show pure intrisic ROI/ITM/playing ability is that only a bot - a la deep blue - can maintain a level of play that follows a "when faced situation x, i will ALWAYS to y."

Humans don't do that. we make mistakes due to various extrinsic factors while we play. These include being impatient, feeling "lucky", alcohol/drugs, an argument we just had, etc. Of course, in poker mistakes can end up being VERY profitable in the ST, I digress...

The point is, we are not bots. Our play varies due to a bunch of factors and relying on an ROI of X means nothing as to how well you will do if you vary your play too much at a given time.


"I think that when Poochie isn't on screen, all the other characters be asking 'where's poochie?'" - Homer.

[bearly]

this is all well and good..but, if your going to talk about endogenous and exogenous factors wouldn't it be profitable to divide your inquiry up into the various size of the buy-ins? the bankroll of the players? the differing nature of 'how the game is played ' at the different levels. hey, pschologism has it's place, especially when all these factors that are being discussed can't be considered in a vacuum. btw, admittedly, psychologism is reductive---but good 'for the sake of the argument.

[FatalError]

so raptor is a bot?

[Blarg]

[ QUOTE ]
I like this discussion. Some thoughts i had...

The reason that I agree only a bot can show pure intrisic ROI/ITM/playing ability is that only a bot - a la deep blue - can maintain a level of play that follows a "when faced situation x, i will ALWAYS to y."

Humans don't do that. we make mistakes due to various extrinsic factors while we play. These include being impatient, feeling "lucky", alcohol/drugs, an argument we just had, etc. Of course, in poker mistakes can end up being VERY profitable in the ST, I digress...

The point is, we are not bots. Our play varies due to a bunch of factors and relying on an ROI of X means nothing as to how well you will do if you vary your play too much at a given time.


"I think that when Poochie isn't on screen, all the other characters be asking 'where's poochie?'" - Homer.

[/ QUOTE ]

I had to pee really bad while playing tonight and was climbing out of my chair when I saw it was my turn on one of my tables. My roll wasn't the biggest, and the time seemed right, so I made a biggish bet with KTo, hoping the get the BB to fold. He did, mission accomplished -- and then I noticed that someone in Seat 5 was also in the hand. I noticed because he re-raised me.

That never would have happened to a bot.

[raptor517]

yes, im a bot. how else could i play 2k+ sngs in one month? holla

[Matt Walker]

It'll take a while for all this uncertainty stuff to sink in, but my initial reaction is that is it impossable to ever separate the two no matter what the experiment is. If you run bot v. bot simulations what are you really measuring? Consider the case with several different bots at the table each with a different style. A type 1 bot will get into the money a high percentage of the time, whereas type 2 will get into the money much less but will get first a good percentage. Wouldn't the intrinsic uncertainty of ROI in the second style and ITM in the first style be bigger? Thus your style of play affects intrinsic uncertainty as you define it in your experiment. Because this varies from style to style and person to person, it is pointless to even consider measuring it.

Also what if we adjusted the mix of bots at the table so there are more type 2 bots. I would guess the uncertainty levels of all bots would change. Thus even for a given level of play the intrinsic uncertainty will vary based on the play of your opponents.

I would venture that no experiment no matter how well devised could measure the uncertainty in poker. There are just too many variables that when you abstract away from yeild meaningless results. IMHO I think looking at data empirically gives as good of an idea as anything.

[Slim Pickens]

You're probably right that it's impossible to separate the two. The experiment would be too complicated. It's also, in effect anyway, a matter of semantics what we call intrinsic and extrinsic, but the discussion is still "there are too many factors to consider, and the statistical uncertainty in only a part of the total."

I'm glad someone noticed the Poochie avitar. It's in honor of my new Empire SN.