Imagine I have a factory for assessing the fairness of coins. I have no assumptions on the coins; i.e, a given coin has an equal probability of exhibiting any form of "bias". For example. the probability of a coin being exactly fair (50-50) is the same as the probability of it being "biased" towards head in a 70-30 ratio.
If I were to toss a given coin a given amount of times (thus creating a sample), over time I would have come up with a binomial distribution for coins with the same bias. Well, that is given a fixed amount of times per sample, but if I were to try and generalize my method and not give a fixed amount, I would come out with a "continuous form of a binomial distribution". Let's call it a "sample distribution".
If I were to try and come up with the sum of PDFs of all of the possible sample distributions (one for each coin bias ratio), I would imagine I would have come up with a U shaped function (much like the PDF of a beta distribution with a=b<1). In a sense, this function would describe the PDF of the distribution of sample results over time for all of the coins coming into my factory in a world where my initial assumption is correct.
If I understand correctly, according to the frequentist paradigm, my confidence level and interval would be the same for any sample result (given that I do not change the sample size), but in a world where my initial assumption is correct, that would mean that I would be wrongly skewed in my estimations of coins; i.e, my estimations would determine that the proportion of bias in coins is far greater than it actually is.
How do I reconcile this?
I believe I have missed a crucial point in the definition of the confidence interval, where it says that the interval's width is not only dependent on the confidence level and the sample size, but also on the sample's variance - the larger the sample variance, the larger the interval width.
In terms of our sample -- a binomial experiment -- the variance of a sample can be intuitively expressed in the ratio between 1's and 0's, which is expected to be lower as we stray from 50%. It also probably important to take into consideration that the binomial distribution for every
p will approach a limit (CLT?), indicating that samples of a fair coin (0.5) should not exhibit small variance with a large enough sample.
This means that the larger the coin bias is (further from 0.5), the smaller its interval width would be (or given a fixed desired interval width, the higher the confidence level would be), which is in fact even "worse" than if the interval would've been fixed to the sample size and confidence level (regardless of sample variance).