0
$\begingroup$

I have a results of sessions where for each $i\in\mathbb N$ the $i$'s session contains $n_i$ Bernoulli trials.
I want to analyze the narrowness of my sample distribution.
From online reading I see some alternative ways to do that.

One is finding the "confidence interval" at , say, 95%.

I calculated this one using the Binomial CDF. i.e.

confidence interval $= (\frac{k_1}{n},\frac{k_2}{n})$

where:

$n$ is the number of trials in the session

$k_1$ is the first $k$ s.t. the probability to be below $k$ successes is below 0.05 and

$k_2$ is the first $k$ s.t. the probability to be below $k$ successes is above 0.95 and what

Another alternative is calculating the standard error of the mean by: $$SEM = \frac{\sigma}{\sqrt n} = \frac{\sqrt{ p(1-p)}}{\sqrt n}$$

So here:

confidence interval $= (mean + SEM , mean - SEM)$

My questions are:

  1. What are the advantages and disadvantages of one method over the other?

  2. Which one of the two would you recommend to use to get a more exact estimation of "how good is my sample mean?"

  3. Are there any other good ways to estimate my sample mean that have advantages over the two methods mentioned above?

  4. Is there a chance that the two methods will give the same interval up to the third after-decimal digit (I didn't check more digits)?
    Because this is what I got using two Matlab functions for the two methods I mentioned. Somehow it seems unreasonable to me that SEM behaves exactly like 95% confidence interval.

Thanks!

a note: as probably shown, I don't have any real background in statistics so any corrections to what I wrote above (if needed) are welcome

$\endgroup$
0
$\begingroup$

The goal of a 95% confidence interval for a parameter, is that if you repeated your data collection infinite times, 95% of the time your confidence interval would contain the true parameter. As you've found out, there are several theoretical ways you could formulate a confidence interval. Ultimately, someone got a PhD somewhere for running millions of computer simulations generating data with a known parameter and working out the proportion of times that formula got it "right" (the true parameter was contained in the interval). This is called the coverage probability.

I think your mean + sem actually needs a mean +/- z sem because I think this is the so-called Normal approximation method. $\hat{p} \pm z \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}$ where for a 95% confidence interval you would use a z value of $1.96$. There are many alternatives to this, such as the Wilson Score interval, the Agresti Coull interval. You can even use an arcsin transformation which fixes the problem of the Normal approximation occasionally giving you intervals which are smaller than $0$ or bigger than $1$.

You could find a comparison of methods in places like this: https://doi.org/10.1002/(SICI)1097-0258(19980430)17:8%3C857::AID-SIM777%3E3.0.CO;2-E My recollection is that the Wilson Score interval is regarded as the "best" and by best I mean it is most likely to give you a 95% CI that has a coverage close to 95% in most scenarios.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.