7
$\begingroup$

Here's the thread I got the idea from: http://www.quora.com/Do-men-have-a-wider-variance-of-intelligence-than-women/answer/Ed-Yong

Basically, this is a model that might be able to explain why there aren't more females in prestigious math/science competitions - it might be a statistical artifact arising from the simple fact that there are far more males than females in math/science. If this model applies, then we may not need to assume that male intelligence has higher variance than female intelligence.

The question I'd like to see addressed: If we assume equal means and equal variances (but different sample sizes), then is the model in the paper still the best model when used for predicting, say, the gender composition of the team of the 5-10 best players? Rather than just the gender composition of the grandmaster?

http://rspb.royalsocietypublishing.org/content/276/1659/1161.full#sec-3 has the diagram and use of the model

They basically used pairing between the top 100 males and top 100 females. Is that a valid assumption to make though? It works for grandmasters - that's true - but would it work if we're trying to select the top 10 people in any field? It's entirely possible, after all, that the expected distributions would be different if we're trying to select from a random distribution of the top 5 players of each gender, rather than the n-th ranked player of each gender.

As you increase the number of players you select for a "winning" team, for example, then maybe the distributions play out in a different way. I would expect the smaller group to have higher variance in mean than the larger group. We know that to be true when averaging over the entire population distribution (as a consequence of the central limit theorem). But what if we just want 10 people from each population instead? The fact is that a lot of "potentially" top people will end up dropping out because they would do something other than spend hours a day to practice for a "winning team"

High variability of the extreme value though - that makes sense if we're talking about the very top. In a large population, the extreme value is going to be very consistent. Whereas in a small population, the extreme value is going to have A LOT of variability - but that extreme value spends far more time in the left part of the (mean of extreme values) as compared to the right part of it. So if you had a head-to-head match up most years, the population with the larger sample size will win.

The thing is, what about a head-to-head matchup of the top 10 members of each distribution? It would be some sort of average between the model the paper used (1 to 1 matchups) and the model where we simply had matchups of the two entire populations with each other.

$\endgroup$
  • 1
    $\begingroup$ Does the problem imply the assumption that the population (sample) sizes are very different $n_{male}>>n_{female}$? $\endgroup$ – Dmitrij Celov Jul 5 '11 at 8:18
  • $\begingroup$ I do not see a clear question here. "Equal means and variances" of what? What precisely is the "model" in the referenced paper (I don't find one)? $\endgroup$ – whuber Jul 5 '11 at 13:28
  • $\begingroup$ Dmitrij: Yes, the problem implies the assumption that the sample sizes are very different. $\endgroup$ – InquilineKea Jul 5 '11 at 17:22
  • $\begingroup$ whuber: Basically, we have two Gaussian distributions whose means and variances are equal. The models are just Gaussian distributions of "talent" or "intelligence" $\endgroup$ – InquilineKea Jul 5 '11 at 17:23
  • $\begingroup$ @Inq But (again) what is the question? By the last paragraph you seem to conclude you don't have a question. The preceding paragraphs refer to a "head-to-head matchup of the top 10 members" in distinction to "1 to 1 matchups": what is the distinction? How do you "match" a group of 10 against another group of 10, if not on a one-to-one basis? $\endgroup$ – whuber Jul 5 '11 at 18:32
2
$\begingroup$

Let's look at the top 3 of 100 Gaussians vs. the top 3 of 1000.
Real statisticians will give formulas for this and more; for the rest of us, here's a little Monte Carlo. The intent of the code is to give a rough idea of the distributions of $X_{(N-2)} X_{(N-1)} X_{(N)}$;
running it gives

# top 3 of  100 Gaussians, medians: [[ 2.   2.1  2.4]]
# top 3 of 1000 Gaussians, medians: [[ 2.8  2.9  3.2]]

If someone could do this in R with rug plots, that would certainly be clearer.

#!/usr/bin/env python

# Monte Carlo the top 3 of 100 / of 1000 Gaussians
# top 3 of  100 Gaussians, medians: [[ 2.   2.1  2.4]]
# top 3 of 1000 Gaussians, medians: [[ 2.8  2.9  3.2]]
# http://stats.stackexchange.com/questions/12647/assuming-two-gaussian-distributions-of-equal-mean-and-variance-then-how-differen
# cf. Wikipedia World_record_progression_100_metres_men / women

import sys
import numpy as np

top = 3
Nx = 100
Ny = 1000
nmonte = 100
percentiles = [50]
seed = 1
exec "\n".join( sys.argv[1:] )  # run this.py top= ...
np.set_printoptions( 1)  # .1f
np.random.seed(seed)
print "Monte Carlo the top %d of many Gaussians:" % top

    # sample Nx / Ny Gaussians, nmonte times --
X = np.random.normal( size=(nmonte,Nx) )
Y = np.random.normal( size=(nmonte,Ny) )

    # top 3 or so --
Xtop = np.sort( X, axis=1 )[:,-top:]
Ytop = np.sort( Y, axis=1 )[:,-top:]

    # medians (any percentiles, but how display ?) --
Xp = np.array( np.percentile( Xtop, percentiles, axis=0 ))
Yp = np.array( np.percentile( Ytop, percentiles, axis=0 ))
print "top %d of %4d Gaussians, medians: %s" % (top, Nx, Xp)
print "top %d of %4d Gaussians, medians: %s" % (top, Ny, Yp)
$\endgroup$
  • $\begingroup$ What is this code supposed to be doing? Exactly what question does it purport to answer? $\endgroup$ – whuber Jul 5 '11 at 13:28
  • $\begingroup$ Thank you for the clarification. If you get a chance, please insert that into your reply for the benefit of future readers. $\endgroup$ – whuber Jul 5 '11 at 15:40
  • 1
    $\begingroup$ BTW the CDF for the max of $N$ iid variates whose CDF is $F$ is just $F^N$. This follows directly from the definitions of CDF and independence. Thus, for any $N$, you can replace the Monte-Carlo simulation with a single (numerical) integration. $\endgroup$ – whuber Jul 5 '11 at 17:07
  • 1
    $\begingroup$ True -- but a) one can Monte Carlo any distribution given by data, e.g. times for the 100-metre dash. b) $F^N$ gives me no feel, no intuition, for the distributions for N = 100, 1000, and imho beginners have to build up intuition; rug plots anyone ? $\endgroup$ – denis Jul 5 '11 at 17:24
  • 2
    $\begingroup$ (a) The formula I gave is completely general; (b) $F^N$ is incredibly simple to plot :-). The mathematics provides intuition far better than the results of a few simulations, because we can generalize from the math using pure reasoning, an approach that is not available from inspecting simulation results. $\endgroup$ – whuber Jul 5 '11 at 18:25
2
$\begingroup$

The earlier answer doesn't address the question of the gender composition of the top $5$ or top $10$ players. The analytical answer is simple, and it doesn't depend on the underlying distribution (as long as it is the same for men and for women, and continuous, and each person's ability is assumed to be independent of anyone else's). Under these assumptions, the number of women among the top $k$ follows a hypergeometric distribution very close to a binomial distribution. If there are $10$ times as many men participating as women, then each spot has a $1/11$ chance to be occupied by a woman. For that question, the techniques used to produce the results in the paper you cite are not needed.

If you want to find the expected value of the $i$th highest out of a sample of size $n$, this is the expected value of an order statistic, and this does depend (slightly) on the distribution. For a uniform distribution on $[0,1]$, the expected value of the $i$th highest value out of $n$ is $\frac{n+1-i}{n+1}$. I think the expected value for an order statistic of a normal distribution doesn't have a closed form in general, but there are good approximations which tell you how to adjust the naive guess of $\Phi^{-1}(\frac{n+1-i}{n+1})$ standard deviations above the mean.

While it may be worth understanding these order statistics as a null hypothesis, I doubt that the distribution of ratings for chess players is so well approximated by the normal distribution that the top ratings out of many millions of players are properly predicted by the corresponding values for a normal distribution. Typically, when you use a normal approximation, you don't count on it working several standard deviations from the mean, and it certainly doesn't work in the other direction.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.