0
$\begingroup$

Imagine I have a factory for assessing the fairness of coins. I have no assumptions on the coins; i.e, a given coin has an equal probability of exhibiting any form of "bias". For example. the probability of a coin being exactly fair (50-50) is the same as the probability of it being "biased" towards head in a 70-30 ratio.

If I were to toss a given coin a given amount of times (thus creating a sample), over time I would have come up with a binomial distribution for coins with the same bias. Well, that is given a fixed amount of times per sample, but if I were to try and generalize my method and not give a fixed amount, I would come out with a "continuous form of a binomial distribution". Let's call it a "sample distribution".

If I were to try and come up with the sum of PDFs of all of the possible sample distributions (one for each coin bias ratio), I would imagine I would have come up with a U shaped function (much like the PDF of a beta distribution with a=b<1). In a sense, this function would describe the PDF of the distribution of sample results over time for all of the coins coming into my factory in a world where my initial assumption is correct.

If I understand correctly, according to the frequentist paradigm, my confidence level and interval would be the same for any sample result (given that I do not change the sample size), but in a world where my initial assumption is correct, that would mean that I would be wrongly skewed in my estimations of coins; i.e, my estimations would determine that the proportion of bias in coins is far greater than it actually is.

How do I reconcile this?

Update

I believe I have missed a crucial point in the definition of the confidence interval, where it says that the interval's width is not only dependent on the confidence level and the sample size, but also on the sample's variance - the larger the sample variance, the larger the interval width.

In terms of our sample -- a binomial experiment -- the variance of a sample can be intuitively expressed in the ratio between 1's and 0's, which is expected to be lower as we stray from 50%. It also probably important to take into consideration that the binomial distribution for every p will approach a limit (CLT?), indicating that samples of a fair coin (0.5) should not exhibit small variance with a large enough sample.

This means that the larger the coin bias is (further from 0.5), the smaller its interval width would be (or given a fixed desired interval width, the higher the confidence level would be), which is in fact even "worse" than if the interval would've been fixed to the sample size and confidence level (regardless of sample variance).

$\endgroup$
10
  • 1
    $\begingroup$ What you refer to as a "sample distribution" sounds like a posterior distribution and your "continuous form of a binomial distribution" would be a Beta distribution. The rules of probability, as embodied in Bayes' Theorem (which a frequentist would use in this setting!), imply the posterior distribution would not be U-shaped. What all this suggests is that you investigate the basic theory of estimation so that you can carry out calculations correctly: that will effect the desired reconciliation. For Web research, good keywords are "conjugate prior," "Beta", and "Binomial." $\endgroup$ – whuber Mar 16 '19 at 14:45
  • $\begingroup$ @whuber I'd rather not delve into Bayesian concepts for the sake of this question, as it seems to me that they add complexity for the purpose of optimization of estimations, which is not an interest of this question. Regardless, I find it hard to understand why such a "sample distribution" would not be U shaped, and what shape would it take otherwise. My reasoning for it being U shaped is that the variance in binomial distribution decrease as we stray from 0.5, meaning that the edges will be more dense, thus having higher values when summing all of the distributions together. $\endgroup$ – Eyal Roth Mar 17 '19 at 8:45
  • $\begingroup$ @whuber (can't edit previous comment) I meant that the sum of all of the "sample distributions" would be U shaped, not that any individual "sample distribution" would be U shaped (they would be reversed-U / bell shaped). $\endgroup$ – Eyal Roth Mar 17 '19 at 9:16
  • $\begingroup$ @whuber Isn't the sum of all the "sample distribution" (what I concluded to be a U shaped function akin to a beta with a=b<1) is just the jeffreys' prior probability for a bernoulli trial -- Beta(1/2, 1/2)? $\endgroup$ – Eyal Roth Mar 17 '19 at 15:07
  • $\begingroup$ You have formulated your situation explicitly in terms of a prior distribution ("I have no assumptions about the coins" really means you are assuming a uniform prior), so there's no avoiding "Bayesian concepts," even though this is a frequentist analysis. I see no way to interpret your "sum of sample distributions" as a prior distribution. $\endgroup$ – whuber Mar 17 '19 at 16:06
1
$\begingroup$

If I understand correctly, according to the frequentist paradigm, my confidence level and interval would be the same for any sample result (given that I do not change the sample size)

Hm, for binomial distributions, your confidence interval depends on p(1-p), so it wouldn't be the same.

Also, with a frequentist approach (e.g., MLE) you don't make an assumption about 50/50, but you simply observe the ratio. I.e., your estimate is "heads / (heads+tails)".

You can make an assumption in the Bayes approach for your 50/50 estimate and do MAP. E.g., your estimate can become

$$ \hat{\theta}=\frac{\left(heads+\gamma_{1}\right)}{\left(heads+\gamma_{1}\right)+\left(tails+\gamma_{0}\right)} $$

where you can add $\gamma_{1}=\gamma_{0}=0$ for no assumption. Or some arbitrary large values if you want to make a strong 50/50 assumption.

$\endgroup$
6
  • $\begingroup$ Why does the interval depend on p(1-p)? $\endgroup$ – Eyal Roth Mar 17 '19 at 8:51
  • $\begingroup$ because the variance of a binomial distribution is np(1-p) $\endgroup$ – resnet Mar 17 '19 at 18:45
  • $\begingroup$ Yes I was missing the part where the variance of the sample is considered when calculating the interval. Take a look at the "update" section of the question. $\endgroup$ – Eyal Roth Mar 17 '19 at 18:59
  • $\begingroup$ Reg the updated question: "This means that the larger the coin bias is (further from 0.5), the smaller its interval width would be (or given a fixed desired interval width, the higher the confidence level would be), which is in fact even "worse" than if the interval would've been fixed to the sample size and confidence level (regardless of sample variance)." yeah, but this is intuitive, right? If you have a 50/50 ratio, there is more randomness (entropy if you will), so the confidence interval should be larger because it's easier to skew your estimate in either direction. $\endgroup$ – resnet Mar 17 '19 at 23:58
  • $\begingroup$ Yep. I guess I had the notion that somehow the frequentist paradigm had an answer to this unfortunate nature of reality. Seems not :) $\endgroup$ – Eyal Roth Mar 18 '19 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.