0
$\begingroup$

Previously I made this post regarding an experiment where people are paired off into groups. For clarity I'll restate the problem here:

Imagine walking into a room with five teenage boys and five adult men who are their fathers. Each child has a unique parent and each father has a unique child. Your job is to guess which men are fathers of which boys based on family resemblance.

Now, imagine that a guesser carries out this experiment once and gets 20% of the matches correct. They carry out the experiment again (on ten new people) and get 80% of the matches correct. A third round with yet again new people results in 60% success. Through each experiment involving ten new people, let's call the number of correct guesses actually made by the guesser $\beta$.

My question is: How do we go about showing that the person doing the guessing is actually seeing the family resemblances and making correct guesses that are better than would be expected by chance? Obviously, a monte carlo simulation would help in this, but is there some statistical test already out there that covers this situation?

This is a different question from my previous post here.

Two points of confusion:

1.) First, this question has been through quite a few revisions, and many of the comments below might be responses to old versions of this question.

2.) For now I'd like to just stick to the $N = 10$ population. At some future point I might expand this to larger $N$, the possibility of siblings present, men and women who might mutually be parents and so forth.

$\endgroup$
7
  • 1
    $\begingroup$ As pointed out in comments to your previous question, this is a (simple) combinatorial problem and scarcely differs from that of your other question: what are the chances that a randomly selected permutation of five objects has, say, three or more fixed points? Answer: count those permutations and divide by the total number of permutations. $\endgroup$
    – whuber
    Commented Jul 6, 2021 at 14:08
  • $\begingroup$ @whuber - I agree with your comment. I've rephrased the question somewhat. I've been having a difficult time trying to figure out how to phrase the question exactly. $\endgroup$
    – the_photon
    Commented Jul 6, 2021 at 14:14
  • $\begingroup$ Your modified question is asked and answered (in full generality) on Mathematics at math.stackexchange.com/questions/1973103/…. $\endgroup$
    – whuber
    Commented Jul 6, 2021 at 14:44
  • $\begingroup$ @whuber - thanks for the link that you provide and I agree that it does address a point I had. I've amended my question again. $\endgroup$
    – the_photon
    Commented Jul 6, 2021 at 14:58
  • $\begingroup$ Isn't the answer the usual "The evidence this person guesses correctly is that they tend to guess correctly"? Or, if you would permit a rephrasing of the question, the evidence that a respondent is not merely guessing is that a random guesser would have a small chance of getting as many (or even more) correct assignments. $\endgroup$
    – whuber
    Commented Jul 6, 2021 at 15:11

2 Answers 2

1
$\begingroup$

The pairing of a father with a son constitutes a permutation from the set of fathers to the set of sons. A fixed point in this problem is when a permutation correctly maps a father to a son.

According to this post, the probability of getting $k$ fixed points by random sampling is

$$P(X=k) = \frac{1}{k !} \sum_{i=0}^{n-k} \frac{(-1)^{i}}{i !} $$

As an R function...

prob = function(k, n){
  s = 0
  for(i in 0:(n-k)){
    s = s + (-1)^i/gamma(i+1)
  }
  
  s/gamma(k+1)
}

You can confirm this with some simulation

# Rerun the simulation one million times
replicate(1e6, {
  # Shuffle the set {1, 2, 3, 4}
  x = sample(1:4)
  # Count how many times the ith number lands in the correct position
  sum(x==1:4)
})->r

# Compute the proportion
prop.table(table(r))

       0        1        2        4 
0.374981 0.333435 0.250176 0.041408 

# Now with our function
> prob(0:4, 4)
[1] 0.37500000 0.33333333 0.25000000 0.00000000 0.04166667

This distribution does not depend on any parameters, so there is no test to perform here. What you can ask is "what is the probability of getting $k$ or more pairs correct", which would be $P(X\geq k)$.

$\endgroup$
4
  • $\begingroup$ Overall, I see your point (somewhat) about there being no parameter to estimate. But in the revised version of my question, isn't $\beta$ the parameter? $\endgroup$
    – the_photon
    Commented Jul 6, 2021 at 14:24
  • $\begingroup$ The number of correct guesses is exactly the thing the distribution describes. What I call $X$ above is what you call $\beta$ (although techincally, $\beta=X/n$ since you express it as a proportion) $\endgroup$ Commented Jul 6, 2021 at 14:33
  • $\begingroup$ In my question, as it stands right now (there have been lots of rapid, small changes to my question), the way I see it, $\beta$ is a number that estimates the number 5, and bounces around up and down from one experiment to the next. Doesn't it follow some probability distribution? I'm wondering what that probability distribution is. $\endgroup$
    – the_photon
    Commented Jul 6, 2021 at 14:33
  • $\begingroup$ Ah! I think I see your point. That $X$ is samples from some probability distribution. I also get your notion that as the question stands right now, there is no separate number estimating any parameter of any other distribution, so as of right now, there's no hypothesis test to do. However, $X$ is still a number that bounces around. From what probability distribution does it come? $\endgroup$
    – the_photon
    Commented Jul 6, 2021 at 14:36
1
$\begingroup$

For the sake of future readers of this post, I want to go ahead and consolidate the answer to this question.

First off, a classical test is what is needed in this case. A classical test is any test where the distribution of the test statistic can be arrived at through direct calculations of possibilities (or other thought experiments). In this particular case, the likelihood of getting $k$ matches correct out of one run through the experiment is given by Demetri's formula above.

A certain amount of computer analysis (that I won't do here) will be needed to determine the standard error of guessing at which boys go with which fathers. As far as I can find off-hand, I haven't seen the exact results of this exact permutation test published anywhere, but neither the distribution nor the permutation test itself are anything truly new. It goes without saying that when a permutation test is used, the statistician may have to construct the distribution de novo.

Once the permutation distribution and its standard error are known, a p-value can be calculated (perhaps through numerical integration). The lower this p-value is, the less likely it is that the guesser is just randomly guessing to arrive at their results in their matching attempts.

I welcome comments if I have mis-stated anything above.

$\endgroup$
4
  • $\begingroup$ Although permutations are involved in the computing the null distribution, that doesn't make this a permutation test! $\endgroup$
    – whuber
    Commented Jul 6, 2021 at 22:07
  • $\begingroup$ @whuber: what kind of test is this then? $\endgroup$
    – the_photon
    Commented Jul 7, 2021 at 2:40
  • $\begingroup$ It is a textbook example of a classical null hypothesis test. It arises as a direct application of the definitions, as explained (for instance) at stats.stackexchange.com/a/130772/919. $\endgroup$
    – whuber
    Commented Jul 7, 2021 at 13:52
  • 1
    $\begingroup$ @whuber: thanks for all your detailed input! I adjusted the wording of the answer. $\endgroup$
    – the_photon
    Commented Jul 7, 2021 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.