5

The term, 'bootstrap confidence interval coverage' is the combination of three concepts: 1. bootstrapping 2. confidence interval 3. coverage probability bootstrapping: The bootstrapping is the resampling method to calculate a statistic. It draws $N$ sets of samples randomly from the original sample with replacement and calculates $N$ statistic from the each ...


5

I'll be honest there: I don't think the actual distinction is all that important. Yes, saying that "the probability of the estimated parameter being included in the confidence interval is 95%" is incorrect, for the precise reason you give. However, I do not think it is a major problem. (I would be interested in any other point of view. Has this ...


3

With only 1000 observations, holding out a separate test set might not be the best approach. See this blog post by Frank Harrell for details. As he says: ... data splitting is an unstable method for validating models or classifiers, especially when the number of subjects is less than about 20,000 (fewer if signal:noise ratio is high). This is because were ...


3

Well, "It's close enough, especially when $n$ is large" In fact, people often do use better approximations when $n$ isn't very large. Some of them are described here on Wikipedia. The Clopper-Pearson interval (described there) always has at least its claimed coverage probability, and is a lot shorter than using p=0.5. There's one setting where ...


3

Let $Y = \sum_{i=1}^{100} X_i$, then we have $Y \sim N(\mu, \sigma^2)$ approximately by CLT. We want to choose $k$ such that $$Pr(\mu-k\sigma \le Y \le \mu \le \mu+k\sigma)=0.95$$ $$Pr(-k \le \frac{Y-\mu}{\sigma}\le k)=0.95$$ We know that $Z=\frac{Y-\mu}{\sigma}$ follows $N(0,1)$. Hence we can read for the corresponding quartile value and find that it ...


3

The standard approach to calculating confidence intervals for odds ratios is to treat them as log-normally distributed. Your data are consistent with this, specifically, In the test group, log parameters $\hat{\mu}_T=3.08$ and $\hat{\sigma}_T=1$ are consistent with an estimated odds ratio of $\exp(\hat{\mu}_T)\approx 21.76$ and a confidence interval from $\...


2

$\psi$ is a smooth function of $\theta$, so the delta-method says $$\sqrt{n}(\psi(\hat\theta)-\psi(\theta))\stackrel{d}{\to} N(0, \sigma^2)$$ where $\sigma^2$ is the limiting value of $n\psi'(\theta)^2\mathrm{var}[\hat\theta]$ You know $\mathrm{var}[\hat\theta]=1/n$, so you just need $\psi'(\theta)$, which is $\phi(0-\theta)$ by the fundamental theorem of ...


2

$CI = \bar{x}\pm t_{n-1, 0.975}s/\sqrt{n}$, at least for the usual $95\%$ confidence interval for the mean. Start by determining the half-width, $ts/\sqrt{n}$, by subtracting the sample mean from the upper endpoint of the confidence interval. Then it’s some algebra to solve for $s$. You can find the value of $t$ from software, such as qt(n-1, 0.975) in R, ...


2

IMHO your best bet is repeated k-fold cross validaton or out-of-bootstrap (possibly also .632+-bootstrap, depending on the actual risk of overfitting). These resampling methods test in turn all cases, and that's the best in terms of CI you can get with such a small data set. They also test an arbitrarily large number of surrogate models, allowing you to ...


2

The BC$_a$ confidence interval uses the percentiles of the bootstrap distribution, but corrects for the bias in the estimate i.e $\hat{\theta}$ as well as estimating the rate of change of the standard error. It is important to note that the BC$_a$ confidence interval adjusts the percentiles, therefore the significance level ($\alpha$) you chose will be ...


2

While I'm not at all convinced balanced accuracy is a useful summary, that's also not how you compute a confidence interval for it. To a reasonable approximation, the estimated sensitivity and specificity will be Normally distributed around the true values. If $$\widehat{\mathrm{sens}}\sim N(\mathrm{sens}, \sigma^2)$$ and $$\widehat{\mathrm{spec}}\sim N(\...


2

This answer takes the same framework as @kevin012's answer, but I'd like to try to be more precise in some of the definitions. I have no trouble with the definition of the bootstrap, or resampling data with replacement and computing the statistic from the Monte Carlo samples. a) Let's say there's as confidence level, $1-\alpha$, and that this is our "...


1

My answer is for small samples of under 20, I agree with the recommended advice of not employing a bootstrap approach. This likely due to the fact that bootstrap does not work well if rare events are missing from the empirical distribution sample (per comment here). Perhaps better is to fit the data to a selected distribution based on knowledge of the likely ...


1

If you truly have data on the entire population of interest, you do not need to estimate anything. You know precisely the proportion in each class because you have measured every individual. However, if what you are truly interested in is the underlying data generating process, you can think of your "population" as simply a sample. Take a look at ...


1

That would be sound. A proportion cannot be negative, so including negative values in a confidence interval does not give meaning, it just looks silly. On the other hand, you could avoid the problem by using methods designed for proportions.


1

What you seem to be missing, is that after sampling the bootstrap sample, you use this data to train the model, then you make predictions using it, and calculate accuracy. Sampling the results calculated on all data would not work, because it does nothing to check what would be the results if you had different data to train your model. See this and this ...


1

It you're asking for work that will be submitted to a particular journal or be reviewed by a particular group, you can always check what other works in that space do, as reporting CIs is a very standard thing to do. What you wrote is fine. For what it's worth, one conventional approach I see is to prefer "X% confidence interval" over "...


1

For three comparisons, you might not get into so much trouble with Bonferroni, but, as another member once put it, the trouble with Bonferroni isn’t that it’s conservative (unnecessarily wide confidence intervals); the trouble is that Bonferroni is ridiculously conservative. Perhaps consider other methods, such as Bonferroni-Holm. But you’re on the right ...


1

Suppose you want to estimate the mean $\mu$ of a normal population using the mean $\bar X$ of a random sample $X_1, X_2, \dots, X_n$ of size $n$ from the population. The term 'standard error' usually refers to the the standard deviation of an estimator. In the current situation the standard error of $\bar X$ is $\sigma/\sqrt{n},$ where $\sigma$ is the ...


1

First of all, if you know that the true mean of the population is 50.06%, you know with 100% confidence that the population mean is within any interval that includes 50.06%. In fact, if you already know the true mean of the population there is no point in making any experiment to estimate it. In second place, if you were actually estimating population mean, ...


1

This isn't really possible to do in sklearn. In order to do this, you need the variance-covariance matrix for the coefficients (this is the inverse of the Fisher information which is not made easy by sklearn). Somewhere on stackoverflow is a post which outlines how to get the variance covariance matrix for linear regression, but it that can't be done for ...


1

If I understand your question correctly, you want to use the inference of the rate from the first experiment to predict the number of successes in the second. This should be possible with pystan. Given an experiment with N trials and S successes and a second experiment of size N2, here is my pystan code: data { int<lower=1> N; int<lower=0&...


Only top voted, non community-wiki answers of a minimum length are eligible