5
$\begingroup$

Supposed I have $n$ coins and I'm interested in finding the $k < n$ coins which have the highest odds of coming up heads and I want to know $p(heads)$ for each of these $k$ coins.

Assume that I'm going to select the top $k$ coins using the maximum likelihood estimate of $p(heads)$ on a finite sample of observations (some for each coin, maybe more for some than others).

After choosing this top $k$, the maximum likelihood estimate of $p(heads)$ seems biased, because we are more likely to choose coins which just got lucky. Our final estimate of $p(heads)$ should probably shrink back towards 50/50, the mean over all the coins, or whatever else is appropriate.

Can anyone suggest a method for getting an unbiased estimate of $p(heads)$ for the coins chosen in this manner?

EDIT: This R code demonstrates the bias I'm talking about

draw = function() {
  n = 100
  k = 5
  m = 200
  probs = runif(n)
  heads = rbinom(n, m, probs)
  top = order(heads, decreasing=T)[1:k]
  err = sum(heads[top]/m) - sum(probs[top])
  return(err)
}
e = replicate(10000, draw())
hist(e)

The errors are not centered around 0.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ You would need to describe the experiment that yields the data on which you make the choice. (There's more than one possible way to generate data that would enable one to make a choice! The most obvious one is simply to toss each coin $m$ times, and choose the top $k$ -- breaking ties arbitrarily perhaps, but there are many other ways one might come to choose the "top" ones) $\endgroup$
    – Glen_b
    Commented Jul 15, 2017 at 2:21
  • $\begingroup$ I could be wrong, but I don't think the process which generates the heads/tails observations matters. Suppose the $i^{th}$ coin had $h_i$ heads and $t_i$ tails. The total number of flips for the $i^{th}$ coin is $m_i = h_i + t_i$. For the sake of making the question easier, let's just say that $m_i = 200$, for every coin. $\endgroup$
    – twolfe18
    Commented Jul 15, 2017 at 12:07
  • 1
    $\begingroup$ Intuitively, the process matters: it's just like what occurs when you engage in a contest with somebody, you lose, and you offer to go best two out of three. That option to choose when to stop, if exercised unilaterally, obviously favors your chances of winning the contest. That suggests that if your experiment were adapted appropriately, it could favor selecting some coins over others. Thus, this isn't just a matter of making the question easier: it's a matter of making it well-defined. $\endgroup$
    – whuber ♦
    Commented Jul 15, 2017 at 14:50
  • $\begingroup$ Yes, I didn't make this clear. I had imagined the observation process as independent/out-of-the-control-of the observer's decisions. The constant $m_i=200$ is only special case of this which I think makes the problem a little easier. I am also interested in more general cases of $m_i \sim Poisson(\lambda)$ (still independent of observers actions). The only observer interaction that I want to stay in the problem is the selection/sorting of coins. $\endgroup$
    – twolfe18
    Commented Jul 15, 2017 at 16:31

1 Answer 1

Reset to default
1
$\begingroup$

If you know the prior then the mean of the posterior distribution of $p$ is an unbiased estimator. In your example with uniform prior, this means that you add 1 tails and 1 heads to the estimates.

That is, if we use the following line in your code

err = sum((heads[top]+1)/(m+2)) - sum(probs[top])

, then the histogram will center more around zero

example with different estimate

(here I have adjusted the breaks and I am using set.seed(1))

In this example the unbiased result depends on the prior information which is not unknown for the example. In practice, the prior might be uncertain, but we could get a reasonable estimate by using the data from all the coins together (for example by fitting a beta-binomial distribution to the data from the coins).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.