1
$\begingroup$

I have two regression models:

Model A:

569 observations 13 predictors

Model B:

569 observations 14 predictors

Model diagnostics are:

R^2 value of Model A < R^2 value of Model B

F-statistic of Model A > F-statistic of Model B

p-value for Model A > p-value of Model B

Residual variance of Model A > Residual variance of Model B

Both models are similarly plausible, physically.

Which model should I pick?

$\endgroup$

2 Answers 2

0
$\begingroup$

Based on this, I would go with B. Of course, I'm sure there is some conceivable situation where A might be better because of some nonlinear cost function, but in general, these characteristics suggest B.

Why?

R^2 is essentially a measure of how much of the variation in the output can be accounted for using the inputs. Higher R^2 means more variation is accounted for with B.

Similarly, lower residual variance with B means more of the variance is accounted for and is thus part of the model.

A lower p-value means it is less likely that the model coefficients are the results of general randomness rather than true system characteristics.

The F-statistic is actually higher for A, which does point that direction (A being a better model). But unless it is massively larger and all others are only slightly skewed in favor of B, I would let this one go, and generally go the other way (i.e. B).

You could make this question a bit more complex by considering a cost function. Is it harder (more time consuming, expensive, etc..) to collect the 14 predictors or the 13? How much does it save you to predict more of the variation? Is the reduction in residual variation worth the increase in collection costs.

$\endgroup$
7
  • $\begingroup$ "A lower p-value means it is less likely that the model coefficients are the results of general randomness rather than true system characteristics." This is incorrect. The p-value only has meaning when compared to a specific null hypothesis and a specific alpha level. Otherwise, except under special circumstance, it is meaningless to compare the numerical value of two p-values. OP has not provided enough information to provide an answer. $\endgroup$ Feb 29, 2016 at 20:23
  • $\begingroup$ but assuming similar null hypothesis and alpha levels, lower p-value would mean lower chance of this. you can complain about what is asked, but unless you're willing to make some level of assumption, you're not going to help anyone $\endgroup$
    – MikeP
    Feb 29, 2016 at 20:30
  • $\begingroup$ OP did not specify the null hypothesis associated with p-value. You cannot assume a similar null hypothesis. In statistics, if you are willing to make some level of assumption without checking it you may not help anyone and in fact may harm them. I don't mind you down voting me, but please comment why my answer is incorrect and how it can be improved. $\endgroup$ Feb 29, 2016 at 20:34
  • $\begingroup$ Tell you what, if the OP comes back tells us what the alphas were and answers all your questions, so that you can give him the full academic support, your suggestions will have been proven useful, and I'll upvote your rant. $\endgroup$
    – MikeP
    Feb 29, 2016 at 20:37
  • $\begingroup$ alpha is simply a critical value of a test. OP needs to specify what the null and alternative hypotheses are that relate to that p-value. That is different than the alpha. I sense some derision in the terms "academic" and "rant", which is a shame. If cursory and misleading advice is what people in your field seek then more power to you. That doesn't fly in my line of work. I am merely attempting to point out to OP and to the wider audience that the metric cited in OPs question are not sufficient for choosing between models, and that there a more appropriate methods for comparing models. $\endgroup$ Feb 29, 2016 at 20:48
1
$\begingroup$

You have not given us enough information to tell you which model should be preferred.

First, it may be that neither model is a good model. An analogy: is an aluminum boat with a big hole in it better than a wooden boat with a big hole in it? Well on one hand, the aluminum boat is lighter, more fire resistant, etc. On the other hand, neither boat floats. Have you checked that both models satisfy the assumptions of linear regression (normality or least symmetry of residuals, no heteroskedasticity or strong outliers, etc.)?

$R^2$ is in general, not a very good metric for choosing among models or even describing models. There any many, many instances in general linear modeling where $R^2$ is misleading or just plain bogus. See Anscombe's quartet for one example. Any time you add a predictor you will get a lower $R^2$, up to adding a predictor for every single data point which will yield a perfect $R^2$ of 1 and a model which is entirely useless for prediction or description.

What are the F-statistic and p-value of? Without more information these are meaningless comparisons. An F-statistics and a p-value relate to a specific hypothesis, so you'd have to specify what that is before we could tell you how to interpret. But in general, the difference between statistically significant and not statistically significant, is not itself statistically significant. This means that there is no meaningful distinction between a p-value of 0.0501 and 0.0499, or between a p-value of 0.001 and 0.002. P-values should not,in general, be treated on a continuous scale or compared to one another. Source.

So in short, none of the information you have given is helpful in determining which model is better description of the data.

Some recommendations:

In general, more parsimonious models are preferred over less parsimonious model. This is rationale behind information criterion statistics that are so in vogue these days. You could consider adding AIC or BIC to the mix of criteria you have listed.

Lastly if these are nested model, i.e. they only differ by the addition of one predictor (you haven't specified, but I'm going to go out on a limb and assume they are) then you can perform a likelihood ratio test. That will tell you directly whether the addition on one more predictor produces a meaningful difference in the descriptive power of the model. Which is better means of testing the hypothesis that these two models are different than a comparison of summary statistics.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.