1
$\begingroup$

Suppose I want to find a linear model with Gaussian error for a given data set. (The data set contains insurance claims and the end goal is to predict claim cost from claim features.) Also, suppose that I use some sort of stepwise model selection method, let's say based on minimizing AIC, to select the variables to include in the model. The main focus is on predictions and interpretation of the coefficients would be a plus.

My question is: when do I check that the model assumptions are valid? before/after or at each step of the stepwise procedure?

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ Stepwise selection is not recommended, especially for interpretation. What kind of data do you have and what is the end goal? $\endgroup$
    – user2974951
    Commented Sep 16, 2019 at 6:17
  • $\begingroup$ @user2974951 the data set contains insurance claims and the end goal is to predict claim cost from claim features. $\endgroup$
    – lalessandro
    Commented Sep 16, 2019 at 6:24
  • 2
    $\begingroup$ Then why are you doing stepwise selection? If your end goal is prediction you may be better off including all the variables, unless you have an enormous amount of them. $\endgroup$
    – user2974951
    Commented Sep 16, 2019 at 12:18
  • $\begingroup$ Could you expand a bit on why this is the case? $\endgroup$
    – lalessandro
    Commented Sep 16, 2019 at 16:09

1 Answer 1

Reset to default
4
$\begingroup$

The basic assumptions of a linear model (independence, homoscedasticity, and normality of errors; also see here) primarily apply to whether inferences about the parameter values are valid (i.e., are the p-values right). For example, ordinary least squares regression is unbiased even if the errors are not normally distributed, or are only uncorrelated (but not independent). Using stepwise selection routines guarantees that your inferences are not valid, however, so it's harder to see what the need is.

These assumptions also don't have much to do with the out of sample predictive accuracy of your model, which you state is what's most important to you. That should be assessed using other means, such as cross-validation. I should note here that stepwise methods are typically quite poor for predictive modeling as well (see, e.g., my answer here: Algorithms for automatic model selection).

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ In addition, a linear model with Gaussian errors is unlikely to be a good way to predict claims payouts, since claims can't be negative & will be zero-inflated. $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 16, 2019 at 20:24
  • $\begingroup$ Sure I understand now. A couple of follow up questions: 1) does this mean that when looking for a prediction focused model all we need to check is that the estimated out if sample prediction error is low? 2) could you give an example of two models, one of which has assumptions validated by the data but poor predictive power and one which doesn't have assumptions validated by the data but has good predictive power? $\endgroup$
    – lalessandro
    Commented Sep 17, 2019 at 6:11
  • $\begingroup$ @alessandro, you should probably ask those as new questions. (1) You may do as you like, but if you are building a model for prediction, it seems to me you would care how well it predicts, not about other stuff. (2) It isn't that having valid assumptions & good predictive ability are mutually exclusive, or even orthogonal. Nonetheless, they aren't the same thing & ultimately you need to decide what you want to optimize: a property you care about, or 1 you don't (cf, When will a less true model predict better than a truer model?). $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 17, 2019 at 17:54
  • $\begingroup$ Again, the point is that the assumptions are needed for valid inference, but if you are using stepwise selection, your inferences are not valid either way. If you feel like checking them just because you have some time to kill, go ahead. $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 17, 2019 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.