I'm using linear regression to predict a continuous variable using a large number (~200) of binary indicator variables. I have around 2,500 data rows. There are a couple of issues here:
- When I run univariate regressions, most of the indicators have significant associations with the dependent variable (by design - I collected data I thought would be associated with the dependent variable)
- Some of the indicators have very low incidence (~1% or less) (but I try to exclude these variables from analysis)
- Some of the indicators co-occur
I'm looking for a stable, parsimonious model. I've been using AIC or BIC selection. I'm aware this can be frowned upon, but I'm comfortable with it, primarily because I'm only feeding in variables I'd expect to have some association. I don't want to use a standard Enter regression because of IDV (independent variable) co-occurrence.
Unfortunately, if I construct a new dataset using random draws of cases from the original dataset, and refit my BIC model, I find that after 20 such loops the majority of the terms each individual BIC model retains are not preserved across all 20 models. I'm thinking I should use some sort of regularized regression - say ridge (for its ability to deal with IDV correlations), lasso (for its ability to set parameters to 0) or elastic net (to combine these advantages). My understanding is that this is akin to using a softer thresholding criteria than stepwise AIC or BIC that can yield more repeatable results. I've never used these techniques before, but after doing some research I'm comfortable selecting a tuning parameter using cross validation to maximise performance in terms of prediction error (though if anyone feels compelled to provide a walkthrough here, I wouldn't object).
My main question: I'm actually more interested in interpreting my model than I am in using it to make predictions. I've read that selecting a tuning parameter for interpretation is harder than it is for prediction, but haven't found any more information. Can someone point me in the correct direction? I'm using R, if anyone is curious.
A secondary question: R provides (I guess Wald) significance estimates for the models in a lasso model. Do I trust them? Can I, at a pinch, interpret them?
I have an engineering background, so I'd particularly like referenced answers (web resources are fine) if possible, and math is welcome as long as its provided with an explanation/intuition.