5
$\begingroup$

I repeatedly read that one can stop looking further when the F-test is insignificant for linear regression; for example in the comments here. However, according to this very insightful answer only relying on the F-test could lead to missing an actual significant variable when some other explanatory variables mask a significant overall result. My question: In case of an insignificant F-test, when shall I stop and when might it be legitimate/worthwhile to continue and look deeper?

$\endgroup$
3
  • $\begingroup$ This probably needs more details to matter at all for any actual data analysis. I think you're viewing statistical testing as a means-to-an-end, and as a geometric process: how many p-values do you need 'til one is significant? No, all hypotheses need to be prespecified prior to data collection and analysis. Post-hoc tests are just that: incidental findings. A rather nice but paywalled paper called "MUNCHAUSEN'S STATISTICAL GRID,WHICH MAKES ALL TRIALS SIGNIFICANT" shows interestingly that with enough variables, you can "grid" an analysis enough to get significant results regardless of truth. $\endgroup$ – AdamO May 4 at 18:43
  • $\begingroup$ I understand and agree.Thanks! (found this also helpful: projects.fivethirtyeight.com/p-hacking) But when my model is theoretically well versed and I have an insignificant F-test. In case I stop, I might miss an actually valid and significant IV, isnt it? Is it a rule of thumb to stop when the F-test is insignificant? $\endgroup$ – mire May 4 at 19:32
  • $\begingroup$ Just wanted to say apologies for the answer that keeps appearing and disappearing! I keep thinking I'm totally content with my answer and then I discover something else. I'm still working through it. Quite fascinating. $\endgroup$ – Phil Assheton 13 hours ago
6
$\begingroup$

As a stats educator I have always told my (unfortunate) students "well, what is the point of looking at the individual $t$ statistics if the $F$ tells us that the overall picture looks just like noise?". So, this question (and @whuber's example) actually kept me awake last night (not even joking), trying to figure out if I've been talking rubbish. And I think I have. The final answer I've arrived at isn't what I've been taught, nor what I've been teaching for the last decade, so if anyone with more experience/education than I can point out a flaw in my thinking I'd certainly be grateful.

Edit: Most of this was done looking at uncorrelated predictors. After taking the post down a couple of days to look at correlated predictors, the picture is a bit more murky, but I still come to the same conclusion. Small section on correlated predictors added right at the bottom.

My Answer

The answer I have reached is the opposite from what I have always thought. I now think the $F$-statistic does not devalue the results of any given significant $t$-test and should not be a barrier to checking the $t$-tests. But, that it can add value to non-significant $t$-tests.

That is to say: don't use the $F$ as a filter for your individual $t$-stats. But do use it to test interesting compound hypotheses.

The key here is that the $F$ has a different goal than the individual $t$-tests and should, I think, be used in line with this. If your goal is to test the predictors individually, then you might do better to just Bonferroni-adjust your $t$-tests-of-interest and look only at those. If your goal is specifically to see whether a bunch of predictors together can predict your dv, then the $F$-statistic has the upper hand and can detect things that your individual $t$-tests (even before corrections) could not.

The fact that ANOVA is so well crafted to pick up combinations of predictors actually directs it away from detecting individual significant predictors and means it can quite naturally appear to contradict your $t$-tests, just because it is asking a different question.

This conclusion came from looking at simulations based on @whuber's example, and a geometric visualisation to help me think this through. In case you're interested, more details follow... Usual caveats apply, that we are usually better informed by estimating effects and their CIs than looking at significance; also these results are what I've found with some preliminary simulations -- but someone might leap in and say I'm all wrong!

The Simulation

I wanted to understand @whuber's example more generally -- rather than one isolated example, so I ran a whole bunch of simulations that ran similar examples 10000 times. I also realised that this specific case (perfectly uncorrelated predictors) can be transformed into a super intuitive geometric interpretation. You can see a simulation along with the geometric interpretation below.

Each row in the table in the figure corresponds to adding an extra predictor to the model -- you can see the beta of the added predictor, and the % significant according to the ANOVA vs the % of cases where at least one of the Bonferroni-adjusted $t$-tests came out significant. So, row 4, for example has 4 predictors, with betas of 0, 0, 0 and 0. Clearly this is the null case, so we're relieved to see lots of 5% values in there.

The plot shows all the simulations that correspond with the second row of the table: where we have two predictors. The axes give the $t$ statistics for the slopes on those two variables; each dot is a simulation. Those dots outside the circle will be significant under the ANOVA, those dots outside the square will have at least one significant Bonferroni-corrected $t$. (The circle was calculated based on the fact that the $F$ in this uncorrelated case is the mean of the squared $t$ statistics). Brightly coloured dots are those simulations that are called significant by only one of our two methods (ANOVA, Bonferroni $t$).

The null case

Now, here is the plot of 10000 simulations based on @whuber's example, with number of predictors set to 10:

Only one meaningful predictor

It really surprised me to find that, in this case, the Bonferroni approach is consistently more powerful than the ANOVA?! But, looking at the graph, we can see why: all those red dots that land in that little "blind spot" of the ANOVA, which is exactly where we'd like it to be.

This held even with a lower effect size and so, lower power (correction to earlier, due to change in approach to the Bonferroni):

Less power: Bonferroni still better!

Where the ANOVA comes into its own is when looking at combinations of predictors. Say, if we add a second meaningful predictor to the model:

Two non-zero betas

...albeit, the Bonferroni lags but is not too too far behind! But still, we can see that, whereas the ANOVA has a blind spot on the "only one meaningful predictor" case, the Bonferronied-$t$ has its corresponding blindspot in the "combination of predictors" case (the cyan dots). And the more meaningful predictors we add to the model, the further and further the Bonferronied-$t$ starts to fall behind:

Ten non-zero betas

So What?

The key takeaway for me from all of this, was that the $F$-statistic is actually somewhat apples and oranges with the $t$-tests. It isn't necessarily the perfect umbrella test, if you want to check predictors one-by-one for significance. It is (I think?) the most powerful option for detecting departures from noise overall: by sitting its circle on the contours of the distribution of dots in the plots, it means the non-rejection region is the smallest possible. But we gain that power in detecting combinations that are very hard for us to interpret.

If you are in a situation where you have ten predictors that all have $t$-statistics of 1.4, it will be able to detect that as "unusual" (a.k.a. significant). But what are you going to do with that? How are you going to then decipher which of those is really behind this? Your $t$-tests aren't going to be much use. If, instead, you use the Bonferroni-adjusted $t$ statistics, you do lose power overall, potentially quite a lot. But you actually gain some power in the situations that you can interpret and that are in line with your objectives. And, crucially, here is the key point: doing so is completely consistent; starting with an ANOVA is mixing together two different questions as if they were one.

So, I think (as it stands at present), I can no longer justify this approach of using the $F$ as a gateway to the $t$ tests.

On the other hand, you might be specifically interested in the combination of a set of predictors. For example, if there is too much noise to detect them individually, or if they don't make sense individually (such as dummies of a categorical variable). In that case, ANOVA really is the stuff!

Code

Here is the code, in case you want to play with it. Sorry it's so long, but I like that you can tweak almost every aspect of the simulation:

library(ggplot2)
library(gridExtra)

# choose your betas to try different possiblities.  You can change the number of betas and it will change the number of xs -- which will also change the dataset size according whuber's approach
betas            <- c( 0,  0, 0, 0, 0, 0, 0, 0, 0, 0)
betas            <- c( 1,  0, 0, 0, 0, 0, 0, 0, 0, 0)
betas            <- c(.5,  0, 0, 0, 0, 0, 0, 0, 0, 0)
betas            <- c(.7, .7, 0, 0, 0, 0, 0, 0, 0, 0)
betas            <- c( 1,  1, 1, 1, 1, 1, 1, 1, 1, 1)*.5

data.multiplier <- 1 # set this at 1 to reproduce @whuber's example.  A higher number adds more data to the dataset (e.g. 2 will double the rows up)

# crude attempt to always start out with 80%ish power for the one-predictor cases
num.predictors <- length(betas)
num.data       <- 2 ^ num.predictors * data.multiplier
noise.sigma    <- sqrt(1/8) * sqrt(num.data) 
num.sims       <- 500 # Number of simulations(reduced to 500 here as better to play with)

set.seed(1)

run.anova.on.subset = function(i, x, y){
  test <- summary(lm(y ~ x[, 1:i]))
  F <- test$fstatistic
  
  num.coeffs <- nrow(test$coefficients)
  p.F        <- 1-pf(F["value"], F["numdf"], F["dendf"])
  p.ts       <- as.numeric(test$coefficients[2:num.coeffs, "Pr(>|t|)"])
  ts         <- as.numeric(test$coefficients[2:num.coeffs, "t value" ])
  
  names(p.F) <- NULL
  
  c(
    t1         = ts[1],
    t2         = if(i > 1) ts[2] else NA,
    Bonferroni = any(p.ts < .05 / i),
    ANOVA      = p.F < .05
  )
}

generate.dataset.and.run.all.Fs = function(){
  # this is @whuber's setup with orthogonal predictors of which only one contributes to y
  x <- as.matrix(do.call(expand.grid, replicate(num.predictors, c(-1, 1), simplify = FALSE)))
  x <- do.call(rbind, replicate(data.multiplier, x, simplify = FALSE))
  y <- x %*% betas + rnorm(n=num.data, mean=0, sd=noise.sigma)
  sapply(1:num.predictors, run.anova.on.subset, x, y)
}
runs <- replicate(num.sims, generate.dataset.and.run.all.Fs())

# convert simulation results into a table of powers and dataframe of ts from each simulation
power.table <- apply(runs, 1:2, mean)[3:nrow(runs),]
power.df <- as.data.frame(t(power.table))

ts.df = as.data.frame(t(runs[, 2, ]))

predictors.df = data.frame(
  beta  = betas,
  Bonf. = sprintf("%.0f%%", power.df$Bonferroni * 100),
  F     = sprintf("%.0f%%", power.df$ANOVA      * 100)
)

# critical values.  f is F transformed to scale of t
Fcrit = qf(.95,2,num.data-3)
fcrit = sqrt(2*Fcrit)
Boncrit = qt((1-.05/(2*2)), num.data-3)

# coords for the circle and square on the plot
Boncrit.df = data.frame(
  t1 = c(Boncrit, Boncrit, -Boncrit, -Boncrit, Boncrit), 
  t2 = c(Boncrit, -Boncrit, -Boncrit, Boncrit, Boncrit)
)
thetas = seq(0,2*pi, length.out=300)
circle = data.frame(
  t1 = cos(thetas) * fcrit,
  t2 = sin(thetas) * fcrit
)

# expand xlimits to fit in power table
xlims <- c(min(c(ts.df$t1, circle$t1)), max(ts.df$t1))
RHside <- xlims[2]
xlims[2] <- xlims[2] + diff(xlims)*.5

plot = ggplot(aes(t1,t2), data=ts.df) + 
  geom_point(aes(colour=interaction(Bonferroni,ANOVA))) +
  geom_path(data=circle, size=1) + 
  geom_path(data=Boncrit.df, linetype="dashed", size=1) +
  scale_colour_manual(values=c("grey", "red", "cyan", "black")) +
  scale_x_continuous(limits=xlims) +
  annotation_custom(tableGrob(predictors.df, theme=ttheme_minimal()), xmin=RHside) +
  theme_classic() +
  theme(legend.position = "none")

print(plot)

Correlated Predictors

Once we start to look at correlated predictors, things get a little more complicated. The Bonferroni approach suffers badly when we have two correlated predictors, as the individual $t$-tests must incorporate uncertainty about "which" of them is the "cause" (please read that word loosely!). The ANOVA, on the other hand can just say, hey, these two look interesting; I award you significant!

But, here's the thing. Every way I look at it, it loops back to the same problem. If anything, the points made in the earlier section are more acute here: the ANOVA is far more powerful, by focussing on detecting scenarios that we cannot interpret (that is, if our goal is to look predictor by predictor). In the figure below, the "cyan" column shows that (in this scenario) we have about 20% of experiments come back significant on the ANOVA, but not on the Bonferroni-$t$ (cyan dots in the figure), so we are stuck trying to interpret them.

And, again, because the rejection region must always cover 95% (ish for Bonferroni) of cases in the null, this increased power on the cyan cases must come at a cost. The cost is that a proportion of cases (up to 6% for this scenario) are red dots -- significant under Bonferroni, but not under the ANOVA. Those are cases that would have been interpretable, but we stopped because of the ANOVA. If we just focus on those cases where we would successfully detect the first predictor with Bonferroni (Bonferroni $t$ True Positives -- "BtTP"), the last column of the table ("BtTPs.blocked") shows that the ANOVA would actually cause us to miss 20% of those in the 10-predictor case!

If you think, hey but you're using the Bonferroni-adjusted $t$ values, that's why you're getting these rubbish results... Firstly, the "FtFP" (F-and-then-$t$ False Positives) shows that using the ANOVA followed by unadjusted $t$s produces at least one false positive in up to 22% of our samples. Secondly, the ANOVA isn't meant to allow us to use non-adjusted $t$s here. Indeed, we hope that the ANOVA will be significant to "allow" us to look at the our true positives on predictor 1. But that then does nothing to control the errors on predictors 2-10.

Correlated predictors

(For full clarity: in the above simulation, each odd numbered predictor has a correlation of 0.6 with the predictor after it. But the rest of the correlations are zero.)

Again, the final point is: this is not anti-ANOVA! Indeed, this suggests that if you have two strongly correlated predictors that sit together and make a nice "thing" -- why not blob them together into one $F$ statistic and test both at the same time? That, as well as the cases from before (dummies and combinations that are hard to detect individually). There are many cases where the ANOVA is far the better tool. It is just a case of picking the right tool for the job. But, when it comes to testing a series of individual predictors of a regression, I don't believe ANOVA is the right umbrella for that job.

New contributor
Phil Assheton is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.