0
$\begingroup$

I have a linear regression model which goes something like this:

$Revenue=c+x1+x2+x3+.......x15$

What I am interested in is two of these predictor variables, lets say x1 and x2, as I want to see which one of them has the greatest impact on the dependent variable (Revenue).

I have produced a linear regression model in R which is as below:

$Revenue=c+x1+x2$

I get an R-squared value of 0.49. I can see from the coefficients, that x1 has a much larger coefficient (5.2), than x2(3.1).

My question is, is this a valid inference to make from the model with a low R-squared? If I add further variables to the model e.g x3, x4, x5, I get a greater rsquared value and thus the coefficient values change. Would the model with the greater rsquared value be better to make comparisons between the two predictor variables, or will the simpler model which contains only the two variables of interest be better?

$\endgroup$

2 Answers 2

2
$\begingroup$

First, R² is a bad statistic if you want to know which model will make the best (out-of-sample) predictions or which model gives you the best inference about predictors. R² will get larger with adding more variables (even if these additional variables are independent and random, having no relation whatsoever with the outcome), but you might just overfit, i.e., fitting noise and sampling variation as if it were informative.

The question which of the predictors is more informative is still underspecified: more important in explaining the data you see (potentially even causally), or most important in making accurate out-of-sample predictions? These are different questions.

I would recommend you first get a valid model which makes sense, judged by a model fit criterion like AIC, WAIC, BIC, or via cross-validation, but also judged by your background knowledge. You will then see whether your model selection procedure retains both variables you are interested in. You can also do explicit comparisons of models with vs. without these variables, for example via likelihood ratio tests or the previously mentioned information criteria.

If your best model contains both variables, then you might want to compare coefficients. Make sure the ones you compare are on the same scale (e.g., normalize them) and take into account their uncertainty (for example via confidence intervals, p-values, credible intervals, ...). But be sure to acknowledge that your conclusions depend on the specified model, which is why this should be the first thing you tackle.

$\endgroup$
0
$\begingroup$

It is fine to do parameter inference when $R^2$ is low. When you do a two-sample t-test (equal variance), you are doing regression parameter inference, and a quick simulation would show you that you have low $R^2$ in many situations where you would be completely comfortable t-testing.

Whether or not you should add additional predictors will depend on your goals. You might be able to explain some variability in the response by adding more predictors. You also run the risk of muddying the interpretation, particularly if you include predictors that are not independent of the two predictors of interest. Further, adding many predictors runs the risk of overfitting and just memorizing the data instead of modeling the phenomenon.

Finally, be a little cautious about just looking at the parameter estimates. Also consider their scales. Interpretations are quite different if your variable is in kilometers than if it is in nanometers. Look at standard errors of the parameters. If neither parameter estimate is significantly different from zero, or if their confidence intervals contain one another, you have scant evidence that the values are different. Perhaps the best way to assess which is more impactful would be to do a linear contrast that would give you a confidence interval for the difference between parameters.

EDIT

I like what stefgehrig says about assessing model fit with a metric other than $R^2$. The trouble with other metrics like the AIC mentioned is that it is hard to judge what a good values is. Adjusted $R^2$ or out-of-sample $R^2$ could be easier to interpret. However, do not fret about using a model with an $R^2$ of $0,30$ or even lower. You might be thinking that explain $30\%$ of the variability in the response is a terrible mode since you grew up thinking that a $90\%$ or so was an $A$ and you want to write $A+$ models, but especially for inference, it is fine not to explain a great deal of variability. As I mentioned in my first paragraph, this is routine in vanilla t-testing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.