Pokernomics

[LittleOldLady]

[ QUOTE ]
[ QUOTE ]
Pick up any of his articles published in peer reviewed journals, and/or his NBER working papers, if you're skeptical of his abilities.

[/ QUOTE ]

I don't know much about peer reviewed journals or exactly how the academic world works, so correct me if my assumption is wrong.

It seems to me that when a study is peer-reviewed it is not as useful if it isn't reviewed by people in the know in that particular industry. The peer reviewers are coming from basically the same background as the originator of the paper and they may not understand the nitty gritty issues that may make the study inaccurate. So the peer reviewers would miss the same issues that the originator may miss also.

For example, in his NFL gambling paper, it is clear that he is not a bettor himself, so he doesn't know the nitty gritty of how the market moves. He states that the market barely moves based on his limited research, but if he was betting and hunting for the best lines (as most pro gamblers do), I think he would start to see that the market moves more than he thinks. I don't expect his peers - other economists that probably aren't heavy sports gamblers - to understand these issues either, so they can't correct him or point to that issue as a possible problem. So in the peer review, all they can comment on is how he converts data into conclusions, but his peers may not know if the data is accurate or useful.

I hope Ed Miller gets involved in this project. It would bring in someone who knows the nitty gritty and thus eliminate this possible problem.

[/ QUOTE ]

King has a point. For example, one section of Levitt's Freakonomics book has to do with predicting what names will become popular in the future. While I have not read Leavitt's book (I have only read about it), it seems that he made a bit of a fool of himself with the name predictions--he in fact predicted that certain names would become popular in the future, when they were already at the top of the charts at the time he was writing. Apparently he didn't know anything about babynaming practices, and never bothered to look at the current status of the names on his list.

For a critique of Leavitt's methodology concerning his babyname popularity predictions see http://www.babynamewizard.com/blog/2...e-onomics.html

So the question arises, is he likely to know anything more about poker than he does about baby names? And if he doesn't, is he likely to consult/collaborate with someone who does?

LOL

[jim leach]

[ QUOTE ]
It seems to me that when a study is peer-reviewed it is not as useful if it isn't reviewed by people in the know in that particular industry. The peer reviewers are coming from basically the same background as the originator of the paper and they may not understand the nitty gritty issues that may make the study inaccurate. So the peer reviewers would miss the same issues that the originator may miss also.

[/ QUOTE ]

I'm a fourth year doctoral student in economics who has never published nor peer reviewed anything, so take this with a grain of salt. But from what I understand, when someone is publishing in an academic journal, the editor of the journal sends the submission to a number of people whom the editor believes can speak to the study. If it's an empirical study, for instance, then the paper may be sent to general econometricians, as well as scholars who have published in that specific area. Levitt publishes in the top general journals, which means that the standards are even higher, and the review process even more rigorous.

There are other phases in the publication process that invite fairly rigorous criticism. Levitt's a research scholar at the NBER, and his work is usually paraded first as an NBER working paper. That means it gets read widely by a lot of people, but there is a lot of self-selection in who reads NBER working papers (unlike peer review, where this is not necessarily the case, or at least, is mitigated since the editor assigns the responsibility, and its an understood thing that people, if they hope to get published in that journal, will comply to an editor's wishes). That is, if you see Levitt's name and he's written something on abortion and crime, you're going to read it (a) if you like Levitt generally, (b) you are interested in fertility and/or crime, or (c) there's buzz about it and you're curious. A lot of problems are caught at this stage, since working papers are shown as a way of soliciting feedback prior to publication. It is not uncommon for a study to stay in working paper form for many years before publication. I think his abortion and crime study sat in working paper form for several years, as it was hotly contested and disputed by attorneys, criminologists, economists and sociologists. Many people, as you can imagine, hated the hypothesis and the conclusion of that study (ie, that legalized abortion was primarily responsible for the fall in violent crime rates in the early 1990s), but the methodology was sound, his question important, and the data fine. Still, Ted Joyce (another economist whose specialty is in abortion and fertility) has published recently in the Journal of Human Resources in which he finds no evidence for this "Roe hypothesis". So who is right? Joyce or Levitt or neither? Social science is kind of aggravating in part because, unlike the natural sciences, we cannot perform controlled experiments. We must, instead, rely on statistics and naturally occuring experiments to help us answer these questions. The question about the plausible link between abortion and crime more or less began with Levitt and Donahue's published article, not ended. It may be that the problems you note with the NFL study are similar in this sense - the question may not be settled, at all. Maybe because Levitt's asking the wrong question, making the wrong inference, using the wrong data - I don't knot. But I digress.

So, there's a lot of ways to get a diverse response rate. The peer review phase is the last phase, and I *think* editors send it to people whom s/he believes can help ferret out any remaining problems that may exist. It's not a perfect process by any means, but it is thorough.

[ QUOTE ]
For example, in his NFL gambling paper, it is clear that he is not a bettor himself, so he doesn't know the nitty gritty of how the market moves.

[/ QUOTE ]

I don't think this is true. From what I have read of him, he does gamble some on horse racing. He apparently grew up around horseracing (I have a grandfather in mind, though I may be just imagining this), and has a fondness for it. I know that he also still bets on horses (read his blog around the time of the Kentucky Derby, as he notes he picked the longshot horse who won).

I seriously doubt he is a gambler, though, in the way that some on 2+2 may mean the term (as in, he is not a professional gambler who earns a substantial portion of his income gambling). But I do think he is solidly familiar with it, for whatever that's worth.

He obviously may still be incredibly wrong in this NFL paper you're talking about; I've never read that paper, and probably won't. It's not a subject I'm much interested in, so never bothered. But isn't this also a working paper, still? Or did it get published? I remember seeing it on NBER a few years ago, or so I thought, so it may not be finished. But again, I may be wrong this. I would strongly recommend that you email the email address on the front page of the article (if it has not been published) with any and all criticisms of the study. I am sure that if there are errors of the sort you've described, that Levitt would like to know of it.

[ QUOTE ]
He states that the market barely moves based on his limited research, but if he was betting and hunting for the best lines (as most pro gamblers do), I think he would start to see that the market moves more than he thinks. I don't expect his peers - other economists that probably aren't heavy sports gamblers - to understand these issues either, so they can't correct him or point to that issue as a possible problem. So in the peer review, all they can comment on is how he converts data into conclusions, but his peers may not know if the data is accurate or useful.

[/ QUOTE ]

First, know that whatever conclusion he comes to is data-driven. So if he is wrong in his conclusion about how this market moves, it may be because the data, as you note, is inaccurate or non-representative in some form, or because the methodology he uses is flawed, or both. Just out of curiousity, do you think that the data is flawed or problematic in some way? Is your criticism, in other words, because the conclusion is counterintuitive or because something empirically seems wrong, or both? Again, I have no idea what this paper says or does, so I'm not defending it.

One last thing, though. As for the possible problems he may run into here, I think the potential problem that he will have to address is one of selection-bias. To what degree are the people who submit their pokertracker stats to him representative of all poker players? My dissertation is on the economics of sex and the family, and so there is one particular study that I keep reading about which reminds me of Levitt's study. The pioneer in sex research, Albert Kinsey, published two books on male and female sexuality in the late 1940/early 1950s. These studies (now referred jointly simply as the Kinsey Reports) are believed to have suffered from selection-bias which skewed the results profoundly. Many of his more controversial findings surrounding the prevalence of homosexuality and infidelity were a result of the non-representative nature of his data. For instance, he polled the individuals who attended his lectures (which were not mandatory) to get their sexual histories - people who probably were not representative of the population. He also sought sexual histories from homosexual friendship networks, prisons, and various detention centers. The agreement among later studies is that he found too much of certain types of behavior (like, for example, extramarital infidelity) because the urn from which he drew his sample was not representative of the population at large.

Since Levitt is asking poker players to select into the project, I'm not sure how he is going to address the potential problem, but it's one that is immediately mentioned whenever I tell anyone that he is working on this study. There are some techniques out there for addressing selection bias, I think, but I'm only casually familiar with them. James Heckman (one of Levitt's colleagues at Chicago) won the Nobel Prize for many things, but one of those things was for his work on selection bias. I think there is a technique that he fashioned, but I know so little about it that I won't comment. But it is called "heckit" if I'm not wrong.

All of this to say - I think that Levitt is a very careful and thorough researcher. The kinds of problems people are worried about on here (ie, that he isn't a gambler, and therefore cannot appreciate the nuances of poker play, and thus will come to incorrect conclusions) are really probably not going to be problems in the end. I suspect that he knows far more about gambling and the game that you know. I'm not saying that his infallible, or incapable of mistakes, but he is transparent in his research, and this will be workshopped considerably, as well as available in working paper form online well before it ever reaches the publication stage. This means, it will be possible to review the study's results, and be a part of the debate as to its merits, well in advance. My saying that "he is sharp and smart" was merely to say that he's not going to make amateurish errors like the ones that I see being worried about on here.

But again, take this all with a grain of salt. What do I know? I just learned I failed my econometrics field exam and have to take it again in three weeks, so I'm not the best source for addressing methodological errors in empirical studies (yet).

[jim leach]

[ QUOTE ]
He states that the market barely moves based on his limited research, but if he was betting and hunting for the best lines (as most pro gamblers do), I think he would start to see that the market moves more than he thinks.

[/ QUOTE ]

Okay, I'm going to read this. You've got my curiousity piqued. I can't download it from home, since I don't have an NBER subscription and thus cannot get it off of SSRN. But in the abstract, it says that "in sports betting, bookmakers announce a price, after which adjustments are small and infrequent." But above, you said, " the market barely moves" and that most gamblers search for the best lines. But that's not the same thing, no? I mean, isn't he just saying that individual adjustment by a bookie is slow and infrequent? Not that there is uniform lines across the market.

[jim leach]

[ QUOTE ]
4. My initial reactions sound disturbingly (to me) like how the baseball old guard reacted to Billy Beane in Moneyball. (Though I think the analogy is very incomplete.)

[/ QUOTE ]

If you're defending Moneyball, then you may have one more reason to doubt Levitt's ability to carry out this study. He appears to think Moneyball is overrated; maybe even incorrect (though I don't know if he goes that far; I never read these entries in his blog, so I'm just passing it along).

This is the first entry:

http://www.freakonomics.com/2005/04/...-of-billy.html

Or go here to see several entries related to Moneyball.

http://www.technorati.com/search/bea...s.com/blog.php

[uuDevil]

[ QUOTE ]
The peer review phase is the last phase, and I *think* editors send it to people whom s/he believes can help ferret out any remaining problems that may exist. It's not a perfect process by any means, but it is thorough.

[/ QUOTE ]
It certainly is not perfect. Even the most prestigious journals make embarrassing mistakes. To give an extreme example, consider the case of Hendrik Shoen, whose fraudulent papers on molecular electronics were published and later retracted by the journal Nature.

Self-selection will be a problem. Losing players are far less likely than winning players to have collected tens of thousands of handhistories.

Despite the potential problems, I look forward to seeing the results of this study. I'm also eagerly anticipating my free book. [img]/images/graemlins/smile.gif[/img]

[King Yao]

I don't know what he knows or doesn't know about baseball, but to say that the A's and Beane are doing a bad job because the expected wins is 83 ... is not on target. 83 wins is only slightly above winning 50% of their games. This looks like an average team, right? But who should we compare the A's against - the Yankees, the Red Sox and the Dodgers? or the Royals, the Devil Rays and the Reds? Clearly the A's success rate should be measured against the latter group because they have similar payrolls as those teams. Whereas the former group can afford to spend 2-5 times as much. The expected records of the latter group were probably around 71 games.

When Levitt doesn't take that into account, it shows that he doesn't fully understand baseball. His conclusion from the data that he presents is fine, but his data is not fully accurate in context.

[djack]

[ QUOTE ]

If you're defending Moneyball, then you may have one more reason to doubt Levitt's ability to carry out this study.

[/ QUOTE ]

Yup. His attacks on Billy Beane do give me pause.

[King Yao]

Scott,

Thank you for your response and explaining a bit about how peer review works. The NBER paper I read was a "NBER Working Paper Series" from December 2002. I have no idea if it was published or not...and frankly from my point of view, it matters not. Apologies in advance for my long and possibly rambling post below.

[ QUOTE ]
...in the abstract, it says that "in sports betting, bookmakers announce a price, after which adjustments are small and infrequent." But above, you said, " the market barely moves" and that most gamblers search for the best lines. But that's not the same thing, no? I mean, isn't he just saying that individual adjustment by a bookie is slow and infrequent? Not that there is uniform lines across the market.

[/ QUOTE ]

On page 1, he writes: "With sports wager, however, market makers (casinos and bookmakers) simply announce a "price" (which takes the form of a point spread, e.g., the home team to win a football game by at least 3.5 points), after which adjustments are typically small and infrequent."

He adds a footnote saying: "In the five days preceding a game, the posted price changes an average of 1.4 times per game. When the price does change, in 85 percent of the cases the line moves by the minimum increment of one-half of a point. Thus the posted spread on Tuesday is within one point of the posted spread at kickoff on Sunday in 90 percent of all games....."

Levitt doesn’t understand how huge a move of a half point or a whole point is in the NFL. It is a gigantic, volatile move. To write: “Thus the posted spread on Tuesday is within one point of the posted spread at kickoff on Sunday in 90 percent of all games....." is like saying the Dow Jones isn’t volatile because 90% of the trading days it moves less than 1,000 points.

A little background on football markets first in case you are not aware:

A pointspread is basically a handicap. A team that is a 3 point favorite is expected, on average, to win the game by 3 points. So subtract 3 points from the favorite (or add 3 points to the underdog) and then you find who won or lost the game as far as the pointspread is concerned. The Patriots were a 7 point favorite against the Eagles in this past Super Bowl. They only won by 3, so that means the winning side of the pointspread wager is the Eagles +7.

Lines have a juice (vig) attached to them, and it is usually -110 either way. This means the bettor has to risk $110 in order to win $100, or any fraction of that ratio ($11 to win $10, $22million to win $20million, etc.). If one were to look at this in terms of percentages, that would be roughly a market of 47.6% bid, and an offer of 52.4%. As a former derivatives trader, I equate this roughly to a market of $47.60 / $52.40 on a stock. As you can see, this market is incredibly wide. If the true price was $51, the market would still be $47.60 / $52.40 as the bookies would just keep the conventional -110 vig on the game. The same market as when the price is $49. There are reasons, which I won’t get into right now, why it is useful for the bookmaker to keep a -110 / -110 line instead of moving that line all over the place based on “minute” changes in the market.

There are some key numbers in the NFL. For example, if a line is the home team laying 3 points, there is around a 10% chance the home team will actually win by exactly 3 points. So now consider what this means when a bookie moves a line from 3 to 3.5. Using the mid-market, if 3 is the true fair price, being able to bet +3.5 on the underdog at no juice means that you will win 55% of the time and lose 45% of the time. That’s a great bet. When a line moves from 3 to 3.5 that’s equivalent to a stock moving from $50 to $55. How often do $50 stocks move $5 up or down in one day? The answer is not often. So a half point or full point move is a very large move. Even when it doesn’t seem like the line has moved, it could have moved but is not visible due to the line he is watching. In recent years, some low-vig online sportsbooks have become successful. At these books, it is possible to see the market move from +100 / -108 to +105 / -113 to -105/-103 in a few minutes based on the action that the book took. Yet, at the books with less competitive lines (almost all the places in Vegas), the customer would never know anything changed. They would have simply looked up at the board and seen -110 / -110, and it would look like nothing was going on. Why is there a difference in these markets? Reasons include: accessibility (you have to deposit cash at the online sportsbooks way in advance before you can make a bet, whereas in Vegas, you can just plunk down $50 from the check you cashed a few minutes ago), legality and the degenerate gambling attitude (for years, Nasdaq market makers would routinely buy or sell shares outside of the real market against their own customers. This is because the customers were not able or did not understand there was a better number for them. This may have changed since I’ve been out of the stock market for almost a decade now.)

It is simple stuff like this that he doesn’t present in the paper, and it is clear to me he doesn’t understand the nitty gritty of the NFL betting industry. The market can be volatile and can move without the layman knowing, even if he’s watching the betting screen. That was the main issue I had with it. There are also other problems with his paper, mainly due to his data that he is working with (he does acknowledge that is a possible problem). His data came from a low dollar season long handicapping contest where the entry fee is $250. Few serious true professional gamblers would use up their valuable time to enter such a contest. So therefore, I find his conclusions irrelevant. I mean, his conclusions may actually be true, but based on the data, he can’t draw some of the conclusions that he did.

But don't get me wrong - I think he does come up with some interesting aspects to football wagering in that paper. Using his data, he is able to figure out some issues that pro gamblers already know and figured out after years of betting - and that is to be commended. But it is the fact that he missed these basic things that are crucial – that’s what makes me wonder if he will make the same mistakes.

In another post, you stated something along the lines of his experience in horse racing. To say that he knows horse racing, and equating that to saying he understands sports betting or poker, is like me saying that a Phd candidate in Biology can be expected to write a legitimate paper in Physics (ok, maybe that’s a bit of an over-exaggeration [img]/images/graemlins/cool.gif[/img] ) . Even among the individual sports, there are incredible differences where one person could be an expert in one and have zero expectancy in betting another sport (I’m thinking specifically of the NFL versus Major League Baseball).

It may seem like I am attacking Levitt, but I’m not specifically. Instead, I’m attacking academics in general. I applaud Levitt for taking on these issues because they are interesting to me, I don’t think Milton Friedman or John Maynard Keynes would bother with NFL gambling markets. I read this paper only because that’s what I’m interested in. I would bet that many other economists would make the same mistakes if not more mistakes. That makes me sound like an arrogant prick, right? It sounds that way to me when I re-read what I typed. This coming from me, who only has a B.S. in Economics and who has barely remembered any of it. Oh well, I believe I am correct, and I have stated my reasons.

[jim leach]

[ QUOTE ]
[ QUOTE ]

If you're defending Moneyball, then you may have one more reason to doubt Levitt's ability to carry out this study.

[/ QUOTE ]

Yup. His attacks on Billy Beane do give me pause.

[/ QUOTE ]

Lol. I was mostly joking in writing that, but I think those posts of his on his blog did get a ton of similar responses to yours. At least, based on the subseqent posts along those lines, that seems the case.

[djack]

It's not just the moneyball/billy beane attack -- where I think he was offbase, because the point was market inefficiences in player valuation -- but other things too. There's also the real estate study, which Arnold Kling critiqued.

I guess my objection is that Levitt seems to think he can just go in to things, learn some basics and make conclusions based on data. Unfortunately, sometimes more knowledge is needed in order to understand what the data is trying to tell you. Levitt should really hire Ed Miller or King Yao, who would both probably be interested and flattered.

(and am I the only one who really wishes King Yao were teaching me to bet sports?)