example

Simpson’s Paradox And Gender Discrimination

One sunny day we arrive at work in the university administration to find a lot of aggressive emails in our in‒box. Just the day before, a news story about gender discrimination in academia was published in a popular local newspaper which included data from our university. The emails are a result of that. Female readers are outraged that men were accepted at the university at a higher rate, while male readers are angry that women were favored in each course the university offers. Somewhat puzzled, you take a look at the data to see what’s going on and who’s wrong.

The university only offers two courses: physics and sociology. In total, 1000 men and 1000 women applied. Here’s the breakdown:

Physics:

800 men applied ‒ 480 accepted (60 %)
100 women applied ‒ 80 accepted (80 %)

Sociology:

200 men applied ‒ 40 accepted (20 %)
900 women applied ‒ 360 accepted (40 %)

Seems like the male readers are right. In each course women were favored. But why the outrage by female readers? Maybe they focused more on the following piece of data. Let’s count how many men and women were accepted overall.

Overall:

1000 men applied ‒ 520 accepted (52 %)
1000 women applied ‒ 440 accepted (44 %)

Wait, what? How did that happen? Suddenly the situation seems reversed. What looked like a clear case of discrimination of male students turned into a case of discrimination of female students by simple addition. How can that be explained?

The paradoxical situation is caused by the different capacities of the two departments as well as the student’s overall preferences. While the physics department, the top choice of male students, could accept 560 students, the smaller sociology department, the top choice of female students, could only take on 400 students. So a higher acceptance rate of male students is to be expected even if women are slightly favored in each course.

While this might seem to you like an overly artificial example to demonstrate an obscure statistical phenomenon, I’m sure the University of California (Berkeley) would beg to differ. It was sued in 1973 for bias against women on the basis of these admission rates:

8442 men applied ‒ 3715 accepted (44 %)
4321 women applied ‒ 1512 accepted (35 %)

A further analysis of the data however showed that women were favored in almost all departments ‒ Simpson’s paradox at work. The paradox also appeared (and keeps on appearing) in clinical trials. A certain treatment might be favored in individual groups, but still prove to be inferior in the aggregate data.

Advertisements

Einstein’s Special Relativity – The Core Idea

It might surprise you that a huge part of Einstein’s Special Theory of Relativity can be summed up in just one simple sentence. Here it is:

“The speed of light is the same in all frames of references”

In other words: no matter what your location or speed is, you will always measure the speed of light to be c = 300,000,000 m/s (approximate value). Not really that fascinating you say? Think of the implications. This sentence not only includes the doom of classical physics, it also forces us to give up our notions of time. How so?

Suppose you watch a train driving off into the distance with v = 30 m/s relative to you. Now someone on the train throws a tennis ball forward with u = 10 m/s relative to the train. How fast do you perceive the ball to be? Intuitively, we simply add the velocities. If the train drives off with 30 m/s and the ball adds another 10 m/s to that, it should have the speed w = 40 m/s relative to you. Any measurement would confirm this and all is well.

Now imagine (and I mean really imagine) said train is driving off into the distance with half the light speed, or v = 0.5 * c. Someone on the train shines a flashlight forwards. Obviously, this light is going at light speed relative to the train, or u = c. How fast do you perceive the light to be? We have the train at 0.5 * c and the light photons at the speed c on top of that, so according to our intuition we should measure the light at a velocity of v = 1.5 * c. But now think back to the above sentence:

“The speed of light is the same in all frames of references”

No matter how fast the train goes, we will always measure light coming from it at the same speed, period. Here, our intiution differs from physical reality. This becomes even clearer when we take it a step further. Let’s have the train drive off with almost light speed and have someone on the train shine a flashlight forwards. We know the light photons to go at light speed, so from our perspective the train is almost able to keep up with the light. An observer on the train would strongly disagree. For him the light beam is moving away as it always does and the train is not keeping up with the light in any way.

How is this possible? Both you and the observer on the train describe the same physical reality, but the perception of it is fundamentally different. There is only one way to make the disagreement go away and that is by giving up the idea that one second for you is the same as one second on the train. If you make the intervals of time dependent on speed in just the right fashion, all is well.

Suppose that one second for you is only one microsecond on the train. In your one second the distance between the train and the light beam grows by 300 meter. So you say: the light is going 300 m / 1 s = 300 m/s faster than the train.

However, for the people in the train, this same 300 meter distance arises in just one microsecond, so they say: the light is going 300 m / 1 µs = 300 m / 0.000,001 s  = 300,000,000 m/s faster than the train – as fast as it always does.

Note that this is a case of either / or. If the speed of light is the same in all frames of references, then we must give up our notions of time. If the light speed depends on your location and speed, then we get to keep our intiutive image of time. So what do the experiments say? All experiments regarding this agree that the speed of light is indeed the same in all frames of references and thus our everyday perception of time is just a first approximation to reality.

How To Calculate the Elo-Rating (including Examples)

In sports, most notably in chess, baseball and basketball, the Elo-rating system is used to rank players. The rating is also helpful in deducing win probabilities (see my blog post Elo-Rating and Win Probability for more details on that). Suppose two players or teams with the current ratings r(1) and r(2) compete in a match. What will be their updated rating r'(1) and r'(2) after said match? Let’s do this step by step, first in general terms and then in a numerical example.

The first step is to compute the transformed rating for each player or team:

R(1) = 10r(1)/400

R(2) = 10r(2)/400

This is just to simplify the further computations. In the second step we calculate the expected score for each player:

E(1) = R(1) / (R(1) + R(2))

E(2) = R(2) / (R(1) + R(2))

Now we wait for the match to finish and set the actual score in the third step:

S(1) = 1 if player 1 wins / 0.5 if draw / 0 if player 2 wins

S(2) = 0 if player 1 wins / 0.5 if draw / 1 if player 2 wins

Now we can put it all together and in a fourth step find out the updated Elo-rating for each player:

r'(1) = r(1) + K * (S(1) – E(1))

r'(2) = r(2) + K * (S(2) – E(2))

What about the K that suddenly popped up? This is called the K-factor and basically a measure of how strong a match will impact the players’ ratings. If you set K too low the ratings will hardly be impacted by the matches and very stable ratings (too stable) will occur. On the other hand, if you set it too high, the ratings will fluctuate wildly according to the current performance. Different organizations use different K-factors, there’s no universally accepted value. In chess the ICC uses a value of K = 32. Other approaches can be found here.

—————————————–

Now let’s do an example. We’ll adopt the value K = 32. Two chess players rated r(1) = 2400 and r(2) = 2000 (so player 2 is the underdog) compete in a single match. What will be the resulting rating if player 1 wins as expected? Let’s see. Here are the transformed ratings:

R(1) = 102400/400 = 1.000.000

R(2) = 102000/400 = 100.000

Onto the expected score for each player:

E(1) = 1.000.000 / (1.000.000 + 100.000) = 0.91

E(2) = 100.000 / (1.000.000 + 100.000) = 0.09

This is the actual score if player 1 wins:

S(1) = 1

S(2) = 0

Now we find out the updated Elo-rating:

r'(1) = 2400 + 32 * (1 – 0.91) = 2403

r'(2) = 2000 + 32 * (0 – 0.09) = 1997

Wow, that’s boring, the rating hardly changed. But this makes sense. By player 1 winning, both players performed according to their ratings. So no need for any significant changes.

—————————————–

What if player 2 won instead? Well, we don’t need to recalculate the transformed ratings and expected scores, these remain the same. However, this is now the actual score for the match:

S(1) = 0

S(2) = 1

Now onto the updated Elo-rating:

r'(1) = 2400 + 32 * (0 – 0.91) = 2371

r'(2) = 2000 + 32 * (1 – 0.09) = 2029

This time the rating changed much more strongly.

—————————————–