Possibly unexamined probability distribution

Question

Previously I made this post regarding an experiment where people are paired off into groups. For clarity I'll restate the problem here:

Imagine walking into a room with five teenage boys and five adult men who are their fathers. Each child has a unique parent and each father has a unique child. Your job is to guess which men are fathers of which boys based on family resemblance.

Now, imagine that a guesser carries out this experiment once and gets 20% of the matches correct. They carry out the experiment again (on ten new people) and get 80% of the matches correct. A third round with yet again new people results in 60% success. Through each experiment involving ten new people, let's call the number of correct guesses actually made by the guesser $\beta$.

My question is: How do we go about showing that the person doing the guessing is actually seeing the family resemblances and making correct guesses that are better than would be expected by chance? Obviously, a monte carlo simulation would help in this, but is there some statistical test already out there that covers this situation?

This is a different question from my previous post here.

Two points of confusion:

1.) First, this question has been through quite a few revisions, and many of the comments below might be responses to old versions of this question.

2.) For now I'd like to just stick to the $N = 10$ population. At some future point I might expand this to larger $N$, the possibility of siblings present, men and women who might mutually be parents and so forth.

Answer 1

The pairing of a father with a son constitutes a permutation from the set of fathers to the set of sons. A fixed point in this problem is when a permutation correctly maps a father to a son.

According to this post, the probability of getting $k$ fixed points by random sampling is

$$P(X=k) = \frac{1}{k !} \sum_{i=0}^{n-k} \frac{(-1)^{i}}{i !} $$

As an R function...

prob = function(k, n){
  s = 0
  for(i in 0:(n-k)){
    s = s + (-1)^i/gamma(i+1)
  }
  
  s/gamma(k+1)
}

You can confirm this with some simulation

# Rerun the simulation one million times
replicate(1e6, {
  # Shuffle the set {1, 2, 3, 4}
  x = sample(1:4)
  # Count how many times the ith number lands in the correct position
  sum(x==1:4)
})->r

# Compute the proportion
prop.table(table(r))

       0        1        2        4 
0.374981 0.333435 0.250176 0.041408 

# Now with our function
> prob(0:4, 4)
[1] 0.37500000 0.33333333 0.25000000 0.00000000 0.04166667

This distribution does not depend on any parameters, so there is no test to perform here. What you can ask is "what is the probability of getting $k$ or more pairs correct", which would be $P(X\geq k)$.

Answer 2

For the sake of future readers of this post, I want to go ahead and consolidate the answer to this question.

First off, a classical test is what is needed in this case. A classical test is any test where the distribution of the test statistic can be arrived at through direct calculations of possibilities (or other thought experiments). In this particular case, the likelihood of getting $k$ matches correct out of one run through the experiment is given by Demetri's formula above.

A certain amount of computer analysis (that I won't do here) will be needed to determine the standard error of guessing at which boys go with which fathers. As far as I can find off-hand, I haven't seen the exact results of this exact permutation test published anywhere, but neither the distribution nor the permutation test itself are anything truly new. It goes without saying that when a permutation test is used, the statistician may have to construct the distribution de novo.

Once the permutation distribution and its standard error are known, a p-value can be calculated (perhaps through numerical integration). The lower this p-value is, the less likely it is that the guesser is just randomly guessing to arrive at their results in their matching attempts.

I welcome comments if I have mis-stated anything above.