# Variable selection for linear regression using robust or least squares estimation

I have a data set consisting of one continuous response variable and about 70 predictors. Using this data, I want to construct a linear regression model. However, I don't know what predictors are worth including in the model, so I'll need to utilize a variable selection method that will allow me to isolate specific response variables. Unfortunately, I've noticed that my data violates a number of assumptions associated with linear regression. Therefore, I'll need to utilize a different estimation than OLS, such as robust or least squares estimation.

When running a linear regression with a different estimation method than least squares, how does one utilize a variable selection method such as stepwise? How can this be implemented in R?

• violate in what way? Aug 23, 2013 at 7:27
• Heteroskedasticity is present, no linear relationship (partial residual plot), and non normal error terms. Aug 23, 2013 at 11:55
• Why would you consider least squares then, rather than say a GLM? Aug 23, 2013 at 15:01
• because my response variable is continuous. Aug 23, 2013 at 15:56
• So... kind of like all the neatly available continuous, non-linear, non-normal, heteroskedastic GLM models then? Gamma, Inverse Gaussian, Tweedie, ... Aug 23, 2013 at 16:05

I suppose in the next advice that you use some method for which R squared is not usefull.

Suppose that you use GLM with maximum likelihood estimation. Then you could use likelihood ratio test as a variable selection method (or more properly as a test for model-size). First you must have a baseline model, for example one which do not have any covariates or only minimal amount of those. You can call that H0 hypothesis.

Next you formulate an alternate hypothesis in a way that model size is larger. You can call this H1.

Next likelihood ratio test is done, first you calculate values for the loglikelihood. LR test statistic is actually distributed as a Chisq with k+p-k degrees of freedom, large statistic means that H0 must be rejected. Value of test statistic is following:

-2LogLik(H0)+2logLik(H1)

p is number of extra parameters in the more larger model.

And there is also AIC, BIC and HQIC information criterias which are all based on log likelihood but penalize from too large model. So they prefer simplicity over more complex models.

There is also generalized method of moments estimation (GMM/GEE) for which there are different tests available in the same spirit as for MLE. If I remember correctly there is a Hansens test which tests for model-size and test statistic is distributed as a Chisq. This comes from the fact that parameters are asymptotically normally distributed and their quadratic transformation follows Chisq - distribution.