Simpson's Paradox

Check out the following cool thing which demonstrates the danger of drawing the wrong conclusions from statistics:


A drug company wants to compare the effectiveness of a new drug to an old drug. It tries the old drug on 40 subjects and the new drug on a different 40 subjects. Here are the results along with percent that improved:


New drug: 20 improved, 20 not improved, 50%

Old drug: 24 improved, 16 not improved, 60%


So a higher percentage of patients improved with the old drug. But wait. Analyze the same results another way:


Men only:

New drug: 12 improved, 18 not improved, 40%

Old drug: 3 improved, 7 improved, 30%


Women only:

New drug: 8 improved, 2 not improved, 80%

Old drug: 21 improved, 9 not improved, 70%


So the new drug caused an improvement in a greater percentage of both men AND women than the old drug, even though it caused an improvement in a smaller percentage of all patients!


This has nothing to do with the small sample sizes used. All the above numbers can be multiplied by a million with the same percentages.


I suspect Mason Malmuth will understand what is going on here.

it has nothing to do with the sample sizes, rather the fact that 2 different groups were used.

The problem here is that the women improve more with either drug than do the men, and a higher percentage of women were given the old drug, while a higher percentage of men were given the new drug. So the new drug actually works better for everyone as the segregated data shows, and the combined data is misleading.


This type of problem can be avoided if the proportion of men and women given each drug is the same. It is not necessary to have an equal number of men and women in the experiment. Note that it is always possible for there to be some other hidden variable which causes statistics to be biased.


This concept can be applied to the issue of alleged discrimination. A company or university may accept a higher percentage of male than female applicants. Yet it is entirely possible that every department within the company or university actually accepts a higher percentage of female applicants. This can occur when a higher percentage of female applicants interview with departments which are more selective than the departments with which the greater percentage of male applicants interview as the following example shows:


Department 1 (less selective):

800 male x 80% accepted = 640 male

200 female x 90% accepted = 180 female


Department 2 (more selective):

200 male x 20% accepted = 40 male

800 female x 30% accepted = 240 female


Totals:

680 males accepted out of 1000

420 females accepted out of 1000

"O.J., where were you around 10:30 that night?"


"I was rushing around, packing for my trip to Chicago."--to police.


"I was sleeping."--to limo driver.


"He was hitting golf balls."--Johnny Cochran, to the press.


Now THAT's paradoxical!!