the following regression models were developed based on the same dataset:

model 1: y=a1x1 + b1x2 + c1

model 2: y=a2x1 + b2x3 + c2

model 3: y=a3x1 + b3x4 + c3

where a, b, and c are the regression coefficients; x1, x2, x3, and x4 are the independent variables. All models include x1 on the first part. How to select the best regression model from these models based on a hypothesis testing approach?

Note: R-square and error terms are very close in all models.

  • $\begingroup$ "Best" for what? $\endgroup$ – Dave Mar 15 at 13:09
  • $\begingroup$ @Dave in terms of accuracy but based on hypothesis testing and not R square or other performance measures because these measures are approximately close for all models $\endgroup$ – Yazan Alatoom Mar 15 at 13:13
  • $\begingroup$ How do you define "accuracy"? Setting aside the issues with "accuracy" as a metric in, say logistic regression, you will not get a perfect match for any observation-prediction pair in the OLS regression you appear to be doing, so all accuracy scores are zero. $\endgroup$ – Dave Mar 15 at 13:15
  • $\begingroup$ @Dave I mean what is the best model for prediction purposes. all models' independent and dependent variables are scale values. for example, how I can say that model 3 is better than model 1 and model 2. $\endgroup$ – Yazan Alatoom Mar 15 at 13:25
  • $\begingroup$ You don't want to use $R^2$, so on what metric would "best" be based? ($MSE$ and $SSE$ are proportional to $R^2$, and in your case, so is $R^2_{adjusted}$, so all of those are off the table. $RMSE$ is just the square root of $MSE$, so $RMSE$ is out, too.) $\endgroup$ – Dave Mar 15 at 13:33

In this very specific situation where your models all add one (different) variable to some baseline model with just the $x_1$ predictor, the highest F-statistic, so the lowest p-value from the F-test, will correspond to the highest $R^2$ and $R^2_{adjusted}$ and to the lowest $MSE$, $SSE$, and $RMSE$.

This gets you the exact same information as comparing the $R^2$ or $R^2_{adjusted}$ values of the three models to see which is highest (equivalent to the lowest $SSE$, $MSE$, or $RMSE$), so there is no need to do any hypothesis testing.

  • $\begingroup$ Thank you @Dave $\endgroup$ – Yazan Alatoom Mar 15 at 13:58

In general there is no way to choose the “best” model for a given problem without knowing something about the particular things you are investigating and the particular hypothesis you have.

In this case because you include X1 in all 3 models I assume that your “main” hypothesis is that X1 is related to Y. If that’s the case, then what you are worried about is that the relationship between X1 and Y might be confounded by some third variable that is correlated with both. But if you include that third variable in your model, then you have addressed this possibility. So in this example the best model is the one that accounts for the confounding variable or variables that you are most worried about. But you can’t figure that out in the abstract.

Let’s say you have a hypothesis that smoking (X1) might cause an increase in the likelihood of heart disease (Y). You should be worried that this relationship might be confounded by other variables that are independently correlated with BOTH Smoking and heart disease. But at the same time you don’t want to control for anything that MEDIATES the relationship between smoking and heart disease - i.e. any other intermediate factor that explains WHY smoking causes heart disease. Given all that, let’s say I showed you three models

Model 1: HeartDiseaseRisk=A1Smoking+B1UnhealthyEating+C1

Model 2: HeartDiseaseRisk=A1Smoking+B1HighBloodPressure+C1

Model 3: HeartDiseaseRisk=A1Smoking+B1GeneticPreDispostitionForHeartDisease+C1

There is no statistical test that can tell us which of these models is “best.” All three might have high R2 values and highly significant coefficients, but only the first one is actually useful in telling us whether smoking leads to heart disease, because it accounts for a potential confounder - we know that unhealthy eating can cause heart disease regardless of whether you smoke or not, and that people who smoke also tend to eat unhealthily. The other models are “bad,” either because they control for a mediator (high blood pressure) or because they control for a variable (genetic pre disposition for heart disease) that probably isn't strongly correlated with Smoking, instead of a variable that is (unhealthy eating). But we can only figure all this out by applying substantive knowledge of the sociological correlates of smoking, and the biological drivers of heart disease.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.