3
$\begingroup$

I have done a series of one-way ANOVA tests. For each test I have calculated the corresponding alternative hypothesis F-statistic and confidence intervals, based on the degrees of freedom involved and the corresponding non-centrality parameter. To calculate the value of the non-centrality parameter for each test, I have used the following formula which relates the mean of the non-central F-distribution with the degrees of freedom and non-centrality parameter (cf. Wikipedia and mathworld.wolfram):

$E[F] = \frac{(\lambda + n_1)n_2}{n_1(n_2-2)}$, where $E[F]$ is the mean of the F-distribution, $n_1$ and $n_2$ are the degrees of freedom, and $\lambda$ is the non-centrality parameter.

The resulting estimates of $\lambda$ are consistent in that if I calculate $E[F(x,n_1,n_2, \lambda)]$, I recover exactly the ANOVA F-statistic estimates.

However when I calculate 95% lower confidence bound using the 5% percentile of $F(x,n_1,n_2, \lambda)$, they are inconsistent with the p-values of the ANOVA test. For the cases where $p<0.05$, the 95% lower confidence intervals nevertheless overlap with the null value of 1 (cf. row 6 of the data table below).

Why this inconsistency? Is it because the distribution of H1 is a non-central F-distribution, whereas the distribution of H0 is a central F-distribution? Does this mean that for the ANOVA F-statistic, confidence intervals are inconsistent with null hypothesis testing?

Here is the table of my results:

F-statistic H0 p-values and 5% lower confidence bound

$\endgroup$
3
  • $\begingroup$ You need to calculate a lower 5% bound. The F test is one sided. $\endgroup$
    – Michael M
    Nov 12, 2023 at 9:03
  • $\begingroup$ I calculated the lower 5% bound confidence intervals. The problem remains (cf. Line 6 of the new table). $\endgroup$
    – treemake
    Nov 13, 2023 at 16:58
  • $\begingroup$ I have added an answer, which is not very good (because it is so long ago where I checked all this). $\endgroup$
    – Michael M
    Nov 13, 2023 at 17:30

2 Answers 2

1
$\begingroup$

You are on the right track in that the central $F$ distribution assumes $H_0$ whereas the non-central one assumes $H_A$, but what you are doing with the latter does not make a lot of sense.

To summarize, you have the following quantities:

  • The central $F$ distribution, which will allow you to calculate how likely it is to get any given $F$-statistic under $H_0$. This is your ANOVA $P$-value.
  • The non-central $F$-distribution, which will allow you to calculate how likely it is to get a specific $F$-statistic under $H_A$. This is used in power calculation, since the cumulative density of this non-central distribution above any chosen critical value of the central distribution based on $\alpha$ is the probability of getting $P<\alpha$ under the non-central distribution.

Expanding on the last point, what you are calculating with your quantiles from the non-central $F$ distribution is essentially the central 95% density of your power function, which does not make a whole lot of sense since an $F$-test is one-sided. More commonly you would look at which point this density crosses some critical threshold, as above, and determine the power from that. In this case you are calculating post-hoc power, which is a questionable concept - plenty of discussion around that on this site. To give an example of such calculation, based on the degrees of freedom in FL1_age the critical value at $\alpha=0.05$ would be $F\approx 2.6928$ under $H_0$, the density of your non-central distribution above that value is about $0.6918$, so that's your (post-hoc!) power.

$\endgroup$
6
  • $\begingroup$ I don't think the second half of your answer is correct. There is a 1:1 correspondence of the classic F test and a lower c.i. of the noncentrality parameter of the F distribution. It is non-trivial, so I have once added it to the {confintr} package in R. In practice, I prefer to think about c.i. for the population R-squared (where there is a similar correspondence). $\endgroup$
    – Michael M
    Nov 12, 2023 at 13:40
  • 1
    $\begingroup$ I've edited my answer as this might have been my overinterpretation of how OP is trying to interpret the non-central quantiles. I would still say that there is no expected relation between the lower quantile (which relates to power as I described) and the value 1, this should be a critical value from the central $F$ distribution instead. $\endgroup$
    – PBulls
    Nov 12, 2023 at 14:33
  • 2
    $\begingroup$ The calculations require numeric optimization, thus you are right that the method in the OP is wrong. The question is old, but if someone is interested, this is the documentation with the reference to Smithson 2003: github.com/mayer79/confintr/blob/main/R/ci_rsquared.R $\endgroup$
    – Michael M
    Nov 12, 2023 at 15:49
  • $\begingroup$ @PBulls, In statistics, it is frequent practice to use the 95% confidence interval on an estimated quantity as a surrogate for a null hypothesis test with alpha=5%. Why does this not seem to work for the F-statistic? $\endgroup$
    – treemake
    Nov 13, 2023 at 17:08
  • 1
    $\begingroup$ @Michael M, By 1:1 correspondence do you mean that the lower CI of the estimated F-distribution is “consistent” with the p-value of the null hypothesis test? Could you explain why the method I used for obtaining the 95% confidence intervals for the estimated F-statistic incorrect? Thanks. $\endgroup$
    – treemake
    Nov 13, 2023 at 17:09
0
$\begingroup$

You need to solve numerically for the non-centrality parameter (NCP) such that your estimate agrees with the corresponding quantile of the F distribution. You can't just plug the estimate as the true value.

This is an aweful thing to do, so I have once added such confidence intervals into the {confintr} package.

library(confintr)

set.seed(1)

x <- seq(0, 1, length.out = 100)
y <- x + rnorm(100)

fit <- lm(y ~ x)
summary(fit) # p-value: 0.001468

ci_f_ncp(fit, probs = c(0.05, 1))
# One-sided 95% F confidence interval for the non-centrality parameter of the
# F-distribution
# 
# Sample estimate: 10.93536 
# Confidence interval:
#   5%          100% 
#   2.48508     Inf 

In this case, the lower 5% confidence bound is slightly larger than 0, which agrees with p < 0.05.

I usually prefer to provide confidence intervals for the population R-squared, which I have a better sense for:

ci_rsquared(fit, probs = c(0.05, 1))

# One-sided 95% F confidence interval for the population R-squared
# 
# Sample estimate: 0.09857415 
# Confidence interval:
#   5%       100% 
#   0.02424821 1.00000000 

In both cases, you have 1:1 correspondence with the classic F-test.

Reference

Smithson, M. (2003). Confidence intervals. Series: Quantitative Applications in the Social Sciences. New York, NY: Sage Publications.

$\endgroup$
6
  • $\begingroup$ The function ci_f_ncp() estimates the confidence interval for the non-centrality parameter. Thus it estimates a family of non-central F-distributions. What I need however is the confidence interval for an estimated F-statistic, which are the quantiles of a single F-distribution. Is there a way to estimate this? $\endgroup$
    – treemake
    Nov 17, 2023 at 15:59
  • $\begingroup$ Would it be correct to assume that Sample estimate of ci_f_ncp() represents the best estimate of the ‘true value’ of the non-centrality parameter. In this case, the lower 5% confidence bound of the corresponding non-central F-distribution should be consistent with the null hypothesis central F-distribution? However this does not seem to be the case either: $\endgroup$
    – treemake
    Nov 17, 2023 at 16:00
  • $\begingroup$ Regarding the first comment: you can estimate parameters, not statistics. Same for CIs $\endgroup$
    – Michael M
    Nov 17, 2023 at 19:05
  • $\begingroup$ Regarding the second comment: I think you are still treating an estimate as the truth. $\endgroup$
    – Michael M
    Nov 17, 2023 at 19:09
  • $\begingroup$ Regarding my first comment, yes you are correct. To provide a more rigorous wording, my objective is to estimate the confidence intervals for the ratio of the between-group and within group variances. This ratio follows an F-distribution with n1 and n2 degrees of freedom. The quantiles of this F-distribution would yield the desired confidence intervals. But how to determine this distribution and hence confidence intervals? Can ci_f_ncp() do it? $\endgroup$
    – treemake
    Nov 17, 2023 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.