1
$\begingroup$

I have a data set consisting of one continuous response variable and about 70 predictors. Using this data, I want to construct a linear regression model. However, I don't know what predictors are worth including in the model, so I'll need to utilize a variable selection method that will allow me to isolate specific response variables. Unfortunately, I've noticed that my data violates a number of assumptions associated with linear regression. Therefore, I'll need to utilize a different estimation than OLS, such as robust or least squares estimation.

When running a linear regression with a different estimation method than least squares, how does one utilize a variable selection method such as stepwise? How can this be implemented in R?

$\endgroup$
5
  • $\begingroup$ violate in what way? $\endgroup$
    – Glen_b
    Aug 23 '13 at 7:27
  • $\begingroup$ Heteroskedasticity is present, no linear relationship (partial residual plot), and non normal error terms. $\endgroup$
    – ATMathew
    Aug 23 '13 at 11:55
  • $\begingroup$ Why would you consider least squares then, rather than say a GLM? $\endgroup$
    – Glen_b
    Aug 23 '13 at 15:01
  • $\begingroup$ because my response variable is continuous. $\endgroup$
    – ATMathew
    Aug 23 '13 at 15:56
  • $\begingroup$ So... kind of like all the neatly available continuous, non-linear, non-normal, heteroskedastic GLM models then? Gamma, Inverse Gaussian, Tweedie, ... $\endgroup$
    – Glen_b
    Aug 23 '13 at 16:05
1
$\begingroup$

I suppose in the next advice that you use some method for which R squared is not usefull.

Suppose that you use GLM with maximum likelihood estimation. Then you could use likelihood ratio test as a variable selection method (or more properly as a test for model-size). First you must have a baseline model, for example one which do not have any covariates or only minimal amount of those. You can call that H0 hypothesis.

Next you formulate an alternate hypothesis in a way that model size is larger. You can call this H1.

Next likelihood ratio test is done, first you calculate values for the loglikelihood. LR test statistic is actually distributed as a Chisq with k+p-k degrees of freedom, large statistic means that H0 must be rejected. Value of test statistic is following:

-2LogLik(H0)+2logLik(H1)

p is number of extra parameters in the more larger model.

And there is also AIC, BIC and HQIC information criterias which are all based on log likelihood but penalize from too large model. So they prefer simplicity over more complex models.

There is also generalized method of moments estimation (GMM/GEE) for which there are different tests available in the same spirit as for MLE. If I remember correctly there is a Hansens test which tests for model-size and test statistic is distributed as a Chisq. This comes from the fact that parameters are asymptotically normally distributed and their quadratic transformation follows Chisq - distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.