0
$\begingroup$

Given the following models with their respective p-value for their coefficient, $R^2$ and $F$-statistics, I was asked to determine which model will give the best prediction based on the information given below.

From what I understand, p-value for each coefficient refers to whether to reject the null hypothesis that the coefficient is zero. Since $x_2$ has p-value very small, it is statistically significant, meaning that $x_2$ should be use in a predictive model.

On the other hand, I notice that all $F$-statistics have very small p-value. But I am not sure what does it implies.

$\endgroup$
6
  • 1
    $\begingroup$ It heavily depends on how these models were found. Can you tell us more about that? Or is it a textbook exercise? $\endgroup$
    – Michael M
    Commented Sep 9, 2019 at 17:17
  • $\begingroup$ It is an assignment given by my instructor. As I am new to Econometrics, I am not sure how to compare different models solely using statistics given above. $\endgroup$
    – Idonknow
    Commented Sep 9, 2019 at 23:40
  • $\begingroup$ This table is not very clear about the meaning (often tables are presented without much information in a legend and that is the case for output from standard statistics packages for which the interpretation is considered well known, but it should not be considered good practice, and also those tables are source of many misunderstandings and wrong interpretations of results). $\endgroup$ Commented Sep 10, 2019 at 2:08
  • $\begingroup$ But anyway, if I had to guess then I would say that the F-statistic is probably relating to an ANOVA model comparison (with a null model, when there is only a constant). Although in that case the F-value would be related to $R^2/(1-R^2)$ (and a factor relating to the degrees of freedom which differes for the models) but I can not discover this pattern. For instance it is strange that the F-statistic for model 2 is a lot higher than for model 1 while the $R^2$ is not much different (and the number of estimated coefficients, relating to degrees of freedom, is the same). $\endgroup$ Commented Sep 10, 2019 at 2:12
  • $\begingroup$ The p-values on all of the coefficients (also on the constant term) indicate that these probably relate to a t-test. (you can also get p-values for coefficients when you perform a F-test with anova or chi-squared test with likelihood ratio, but then this does not work for the constant term) $\endgroup$ Commented Sep 10, 2019 at 2:26

1 Answer 1

6
$\begingroup$

In-sample information like what you are presenting has only a very weak relationship with out-of-sample predictive performance.

For instance, a larger model, like your Model 6, will always have a larger $R^2$, i.e., proportion of variance explained, than a smaller model. But that may well be due only to overfitting. And come with worse predictive performance than the smaller model.

Better: hold the last couple of observations out from the model fitting step. Then predict into this holdout sample. Check which model performs best on the holdout sample, using your preferred accuracy measure. Choose the best performer. (Bonus: assess whether the difference in accuracy is statistically or economically significant.)

$\endgroup$
8
  • $\begingroup$ Upvoted for the clarity and usefulness of the answer. $\endgroup$ Commented Sep 9, 2019 at 18:34
  • $\begingroup$ How about their F-statistics? Does it tell us anything? $\endgroup$
    – Idonknow
    Commented Sep 9, 2019 at 23:38
  • $\begingroup$ But don't the low p-values indicate that the model is a significant improvement and not just fitting random stuff? If the model variables would not make sense and are just random then you would not be able to increase the explained variance $R^2$ by that much (at least not more than 1 in 10 000 times). So model 5 seems very good, even without cross-validation. $\endgroup$ Commented Sep 10, 2019 at 2:22
  • 1
    $\begingroup$ @MartijnWeterings: we don't know. These could come from models guided by theory, with a clear rationale for each predictor, in which case I would trust Model 5 most (but still stand by my answer). Or the models could come from someone throwing data at the problem at random and cherry-picking the best-looking $F$ statistics through spurious correlations. In which case I would not trust either one of the models. We don't know. $\endgroup$ Commented Sep 10, 2019 at 6:48
  • $\begingroup$ Yes that is right, but then the p-values should be corrected. Sure more information is needed. My main point was that p-values could be used to select a model. $\endgroup$ Commented Sep 10, 2019 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.