4
$\begingroup$

Can someone interpret me the line in the bold ? how it is different from p-value as p-value is also responsible for the same 5% error ? The text is from "An Introduction to Statistical Learning with Applications in R" by Trevor Hastie and Robert Tibshirani

Given each individual p-values for each variable, why do we need to look at the overall F-statistic?

After all, it seems likely that if any one of the p-values for the individual variables is very small, then at least one of the predictors is related to the response. However, this logic is flawed, especially when the number of predictors p is large.

For instance, consider an example in which p = 100 and H0 : β1 = β2 = ...= βp = 0 is true, so no variable is truly associated with the response. In this situation, about 5% of the p-values associated with each variable (of the type shown in Table 3.4) will be below 0.05 by chance. In other words, we expect to see approximately five small p-values even in the absence of any true association between the predictors and the response. In fact, we are almost guaranteed that we will observe at least one p-value below 0.05 by chance!

Hence, if we use the individual t-statistics and associated pvalues in order to decide whether or not there is any association between the variables and the response, there is a very high chance that we will incorrectly conclude that there is a relationship.

However, the F-statistic does not suffer from this problem because it adjusts for the number of predictors. Hence, if H0 is true, there is only a 5% chance that the Fstatistic will result in a p-value below 0.05, regardless of the number of predictors or the number of observations

$\endgroup$
7
  • 5
    $\begingroup$ If you're quoting a text, you need to give the details of the thing you're quoting (and properly mark the quoted parts as quotes). $\endgroup$ – Glen_b Mar 29 '16 at 12:53
  • $\begingroup$ @Geln_b new to this. Hope will take care of things next time, $\endgroup$ – learner Mar 29 '16 at 13:06
  • 1
    $\begingroup$ It's important to take care of things this time.... $\endgroup$ – Nick Cox Mar 29 '16 at 13:18
  • 1
    $\begingroup$ @user110244 You can edit your question to include the title of the book and its authors. $\endgroup$ – Peter Mar 29 '16 at 14:20
  • 1
    $\begingroup$ @Peter thanks for suggestion. I have done what is asked. $\endgroup$ – learner Mar 29 '16 at 16:34
4
$\begingroup$

Reputed book? The example was somewhat conflated because it's trying to explain the shortcoming of multiple t-tests and yet using an F-test to describe type I error. It's slightly confusing.

Basically, suppose there are 5 groups and each group has a mean. We're interested to see if any of the groups has a mean that is different from others. There are generally two perceivable approaches:

First approach, use many t-tests to compare each mean against 0: G1 vs. G2, G1 vs. G3, G1 vs. G4, G1 vs. G5, G2 vs. G3, G2 vs. G4, G2 vs. G5, G3 vs. G4, G3 vs. G5, G4 vs. G5.

Totally there are 10 t-tests. Just as a kink of hypothesis testing, there is always a chance we will declare a difference even in reality there isn't one. This is called type I error. We often set this at 5% as a convention. Now, there are 10 tests, and each of them individually can have a 5% type I error rate. Chance is you will see some differences that are just due to randomness, and erroneously conclude there some means are different. In other words, using many t-test can inflate the chance of committing type I error, making it bigger than 5%.

The second approach is using an F-test, in this case an ANOVA. ANOVA tests collectively if any of the means is different from the rest. In a way, it groups all the t-tests together into this one big null hypothesis:

$H_0: \mu_1 = \mu_2 = \mu_3 = \mu_4 = \mu_5$

Now, because it's still one single test, it does not face the problem like the ten t-tests have. And because of this, using an F-test instead of running many t-tests is a safer way to guard against mistakenly rejecting any null hypothesis (aka committing a type I error.)


Now, the implication can be broad. For instance, it's applicable to regression models. In regression model, it's crucial to examine the ANOVA results (aka the F-test result) of the model as a whole before diving into the p-values for each independent variables. If the overall F-test says none of the independent variable's slopes is different from 0, then even one of the independent variables has a p-value less than 0.05, we should not interpret it as if it's a significant independent variable.

$\endgroup$
4
  • 1
    $\begingroup$ +1 but can one really use an ANOVA to test the null that all means are equal to zero? Usually the null is that all means are equal between each other but not to some fixed value. $\endgroup$ – amoeba Mar 29 '16 at 13:38
  • $\begingroup$ I have rejected an attempted edit in which it was implicitly assumed the ten t-tests in the first approach are independent--which they definitely are not. This mistake, though, is of interest because it suggests a deeper analysis of your claims would be helpful in getting the point across. $\endgroup$ – whuber Mar 29 '16 at 18:00
  • $\begingroup$ @whuber Can't be 10 t-test give compound error (.95)^10 and leads to alpha = 1-(.95)^10= 0.40126. Am i right ? $\endgroup$ – learner Apr 28 '16 at 11:05
  • $\begingroup$ @user110244 To be valid, your calculation requires independent results. If those t-tests share data, such as a common control group, then they are not independent. This is the problem solved by the F test. $\endgroup$ – whuber Apr 28 '16 at 15:13
2
$\begingroup$

If you have only one variable in the model, then t-test on a single variable and its p-value will be the same as F-test on an entire model and its p-value.

The difference is when you have more than one variable. In this case it is possible to have individual variable's p-values (from t-test) not significant, yet F-test p-values significant. This could point to potential multicollineraity issues, for instance, i.e. maybe you need to kick out a few variables from the model.

$\endgroup$
2
  • $\begingroup$ ok but performance wise both are having 5% error . correct ? $\endgroup$ – learner Mar 29 '16 at 13:10
  • 1
    $\begingroup$ $\alpha=5$% is the significance level which you set before running the tests. It's up to you which $\alpha$ to set in both F- and t-test. It makes a sense to use the same $\alpha$, of course. If you set the same significance, and have only one variable in the model, then both tests will produce the same result. $\endgroup$ – Aksakal Mar 29 '16 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.