17

The AIC is not an estimator of a true parameter. It is a data-dependent measurement of the model fit. The model fit is what it is, there is no model fit that is any "truer" than the one you have, because it's the one you have that is measured. But without any true parameter for which the AIC would be an estimator, one cannot have a confidence ...


12

AIC estimates $-2n \ \times$ the expected likelihood on a new, unseen data point from the data generating process (DGP) that generated your sample.* Even though the target (the estimand) is not a parameter, it is a meaningful quantity. E.g. it may be interpreted as the expected loss of a point prediction. It is quite natural to wish for a confidence interval ...


6

The standard error that you want is the standard deviation of the estimate $\hat\pi$ at a fixed point in the sequence, over multiple experiments. This standard error will give you a confidence interval that includes the actual value $\pi$ in 95% of experiments. You don't need a huge sample size to get reasonable estimation of the standard error; you need a ...


5

So let's start by breaking down the formula for confidence intervals (CIs), Let's consider this in a little more generality then in your question. Let's say we want a confidence interval for an estimator of some parameter of the population, $\hat\theta$ (this could be the sample mean, or a regression coefficient, etc.). The population (true) parameter will ...


5

When you use a paired T-test, you are essentially doing a one-sample test, where your one sample consists of the paired differences between outcomes in two groups. If you create a new sample of these difference values and then apply the formula for a one-sample T-test, you will see that this is equivalent to the paired test.


4

"But what if someone were to switch from a frequentist to a Bayesian viewpoint after constructing the CI and ask themselves the question: "How confident am I (i. e. at what rate would I - given that there is someone who knows μ and will reveal it at some point - be willing to bet) that this given CI contains μ knowing that it was constructed using ...


4

For so called location models, such as your linear regression, anovas etc., basically for models where the outcome depends linearly on the estimated parameters, the confidence interval will be the same as the credible interval with flat prior. If you want to know how would that credible interval look like with a different prior, then you add that prior ...


3

f is the frequency, i.e. the number of students in that category. % is the percentage of students in that category. For example, dividing $3$, $51$, and $6$ by their sum gives you $0.05$, $0.85$, and $0.1$, respectively.


3

Your question actually motivated me to writeup a recently published Stanford paper - link. Here's the writeup if you're curious. Note that you'll still see the performance hits because it uses NCV, but this method seems statistically robust. Finally, I think Cross Validated was missing info about computing the SE because the method didn't exist. Dependence ...


3

"M and SM are quite close and so, M can be replaced with SM." No, M is nowhere replaced by SM, and you don't need to assume they are close. The CLT makes a statement about the distribution, i.e., the variation of the SM, its distance from M. It doesn't require anywhere that this distance is small (although it is expected to become small for large n ...


2

Note that a hazard ratio (HR) estimate of 8 with a top 95% confidence limit of 50* isn't quite as bad as it might seem. Cox regression coefficients are estimated in the log-hazard scale. A quick calculation suggests that corresponds to a regression coefficient of 2.08 with a standard error of 0.93. On that scale, things don't seem so extreme. Your model, ...


2

Comment continued: In case you want to check your answer against computer printout for a Welch t test in R, here are fictitious data and relevant computer printout. set.seed(2021) x1 = rnorm(30, 50, 7) summary(x1); length(x1); sd(x1) Min. 1st Qu. Median Mean 3rd Qu. Max. 36.54 47.66 51.10 51.06 57.38 62.11 [1] 30 [1] 7.651895 x2 = ...


1

To get the value, they subtracted group A from group B (B - A). If B is bigger, you'd get a positive number. If A is bigger, you'd get a negative number. Here, they subtracted stress intensity at the end of 8 weeks from stress intensity at the beginning. Stress went down, so you have a negative number. If stress went up, you'd have a positive number. If your ...


1

As posted in the comments, this refers to an $F$ distribution with $1$ and $78$ degrees of freedom. In the derivation of the $F$ distribution, we divide a $\chi^2$ variable by another $\chi^2$ variable: $$F(a, b) = \dfrac{\chi^2_a/a}{\chi^2_b/b}$$ Here, $a$ and $b$ are the degrees of freedom of the $\chi^2$ variables, and those values make their way to the $...


1

The grey band is a 95% confidence interval for the regression line. It does not say anything about points lying within or outside of the grey area - but you can visually see, whether the upper or lower limit of the regression intervall both show an ascending or descending trend. If so, the slope is significant. For an example with a non-significant slope try ...


1

With an exponential distribution, $\log S_A(t)= -\lambda_A t$ and $\log S_B(t)=-\lambda_B t$. The hazard ratio ($\text{HR}$) you request is: $$ \frac{\log S_A}{\log S_B} = \frac{\lambda_A}{\lambda_B},$$ and the $\log \text{HR}$ is: $$ \log \text{HR} = \log \lambda_A - \log \lambda_B.$$ The confidence interval (CI) for the $\text{HR}$ is related to the ...


1

No, although a lot of confidence intervals you may encounter are motivated by either transforming some statistic onto a scale in which the sampling distribution is something close to normal or by using asymptotic arguments (e.g. with enough data, the distribution is close enough to normal). The sample mean for an exponential random variable is a good example ...


1

Comment. Here is one kind of test you could do in R, illustrated with your fictitious data. Are proportions of proposals submitted by (known) whites changing significantly over time? You give proportions of whites as p.w and numbers of proposals p.w = c(23,24,25,23,27,29,32)/100 n = c(9,10,10,12,13,13,16)*100 So counts of proposals by whites are as follows:...


1

My two comments from above: A confidence interval (CI) is about a parameter (in your case the population mean), not about a sample. So it makes no sense to ask "how many % of the sample is within a CI". The CI tells in what range the population mean might fall with high certainty. Where e.g. 95% certainty means: if you would repeat the same ...


1

From what you say in your Comment, I am not sure what the distribution of R0 might be. Certainly, I see no reason to suppose the distribution would be symmetrical. Suppose you have $n = 30$ observations as in my fictitious data vector x, summarized below: summary(x) Min. 1st Qu. Median Mean 3rd Qu. Max. 0.03995 2.73616 7.92256 9.38491 16....


1

Assuming that the test sets are independent and come from same distribution, the solution is pretty simple: combine the bootstrap samples from all the trials and calculate the interval using the combined samples (i.e. quantiles of the combined samples). It simply sounds like all your bootstrap samples come from the same distribution, so there’s no reason why ...


1

In a two-sample test, you have two independent samples. Of course, by independence, we expect the two sets of measurements will not be correlated. Sample sizes for the two samples need not be equal. (But it often makes sense for them to be approximately equal.) In a paired test you have one sample of pairs. Typically, there will be $n$ paired observations $(...


1

A paper was recently published out of Stanford (2021) that describes how to account for dependence between our data folds. The TLDR is: We use nested cross validation to develop a systematically biased model. This provides an estimate of bias inherent to our data splits. We subtract the bias from a regular CV model, creating unbiased confidence intervals ...


1

If you can't assume independence of the data splits (which in many scenarios you can't), here's a method that allows for the computation of "valid" confidence intervals around your error. It was recently published by Stanford (2021) so there still aren't python packages, but they did create an R package. I was interested in the topic so I made a ...


Only top voted, non community-wiki answers of a minimum length are eligible