# Tag Info

17

As mentioned by @DemetriPananos, theoretical justification would be the best approach, especially if your goal is inference. That is, with expert knowledge of the actual data generation process, you can look at the causal paths between the variables and from there you can select the variables which are important, which are confounders and which are mediators....

11

If the goal is prediction then your proposed solution sounds agreeable, but inference is a whole other beast. The fact is that the theory, for example concerning sampling distributions of coefficients, is not capable of conditioning in such model selection procedures. Frank Harrell talks a little bit about this in Regression Modelling Strategies in the ...

5

Some preliminaries: In LASSO models, the number of non-zero coefficients is an unbiased and consistent estimate for the degrees of freedom of the lasso (see, Zou et al. (2007) "On the "degrees of freedom" of the lasso" for more details). In ridge models, the degrees of freedom are directly related to the singular values of the centred input matrix $X$ (see, ...

5

Eliminating variables is only one of several methods of dealing with collinearity. Others include (the list may not be exhaustive): Getting more data Principal components regression Partial least squares regression Ridge regression Elastic net Which of these is best (or if any is even necessary) depends on your goals and your particular situation.

4

Theoretical justification beyond all else. Aside from that, LASSO or similar penalized methods would be my next suggestion.

3

Sounds like regularization models could be of use to you (Lasso especially, since your problem seems to be centered around picking variables, as opposed to Ridge and Elastic net). However, these can be difficult to set up. They are included in Stata 15 if you have access to that. In R, it's often a bit more handywork but hopefully someone can chime in with ...

2

You have too few observations to include in your initial model that many predictors. Also, note that for binary data the effective sample size is determined by the minimum of the frequencies of the zeros and the ones. Hence, you have very little information in your data to obtain any meaningfully stable results. Finally, as noted in the comments by EdM, ...

Only top voted, non community-wiki answers of a minimum length are eligible