2
$\begingroup$

I'm working on a linear model with the "abalone" data set in R. I tried to build a more traditional linear model but found that the normality assumption was badly violated, even with a natural log transformation of the outcome. I instead decided to try bootstrapping. I ran 2,000 models, each time selecting features using stepAIC() and stepVIF(). I then analyzed which features "survived" within the 2,000 resulting models. Most of my predictors were present either ~100% of the time or ~0% of the time. One predictor however, was present ~28% of the time.

  1. If a traditional linear model does not appear to work, is pivoting to a bootstrap technique a logical next step? Or at least not a ridiculous one?

  2. Is this a feature selection process that makes sense? The other methodology I was thinking of was to run all of the models for all of the predictors, then only retain the ones that have confidence intervals entirely above or below 0. (as determined by quantiles of coefficients)

  3. Assuming this is a valid feature selection process, what should I do about the predictor that shows up ~28% of the time. My instinct is to leave it out on the basis of a naive ~50% threshold, but somehow that feels off.

  4. I also notice that this method gives one predictor labeled "Diameter" a negative coefficient related to the outcome ("Rings") even though they are positively correlated in the full data set. That seems like a red flag, right?

I know this is a long one and I appreciate the support.

$\endgroup$

1 Answer 1

1
$\begingroup$

First off, if you plan to use your model to predict "rings" (rather than infer causal effects of covariates) you would want to consider out-of-sample MSE to make decisions about which model to use. If you're trying to understand relationships/effects of covariates, you should consider which types of confounding are likely to exist in the situation and use that to inform your decisions. I took a look at the dataset description, and I'll assume that you just care about prediction.

  1. From what I understand, using stepwise regression to do model selection is a fairly common practice. Considering multiple iterations of the algorithm is definitely better than running it once and using those results. However, you might also try using ridge regression (one package for this is glmnet) because the variables in this dataset will be fairly collinear.

  2. Stepwise selection is probably a better method than merely choosing coefficients significantly different from zero. (However, ridge regression is probably better than both of those in the presence of multicollinearity)

  3. If you go ahead with stepwise variable selection (and assuming that you're only interested in prediction) you could compare the out-of-sample MSE for the model with and without the variable that occurs 28% of the time, and choose the model with smaller MSE.

  4. The discrepancy between sign of the correlation between a variable and the outcome and sign of the regression coefficient is due to multicollinearity.

I haven't fit any models, but here's a possible example. Imagine a case where abalone get more shell weight as they get older, but their non-shell weight gets big when they're fairly young and then decreases as they get older. In this situation, the total abalone weight might be positively associated with rings/age only because total weight includes shell weight. If you were to fit a model that contained both shell weight and total abalone weight, the coefficient for shell weight would be positive because it strictly increases as abalone get older. However, the coefficient for total abalone weight would represent the effect of total weight after conditioning on shell weight. So, the remaining effect would be negative because non-shell-weight decreases as an abalone gets older.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.