4
$\begingroup$

I have almost no experience with math or stat, but I am trying to run an Adaptive LASSO on a continuous outcome with around 200 cases and a list of around 19 variables. Some of these variables have 3 categories. My questions are:

  1. Is the sample large enough to use an adaptive lasso?
  2. Do I necessarily have to have the train and test data? or I can run the adaptive lasso on all the 200 cases?
  3. Also, my other question is on the variables with more than 2 categories. How does adaptive lasso interpret that?
  4. How can we get confidence intervals for coefficients? Does that even make sense?
$\endgroup$
7
  • 1
    $\begingroup$ Why do you have confidence interval in the title? $\endgroup$ Commented Mar 26, 2022 at 16:59
  • $\begingroup$ If you have almost no experience with statistics, why are you trying to run LASSO? Will you try to interpret your results? $\endgroup$
    – qwr
    Commented Mar 27, 2022 at 7:03
  • $\begingroup$ @RichardHardy Oh, I forgot to add the relative question about it. Because I saw that LASSO doesn't give us a confidence interval as an output and my question was how can we get a confidence interval if that even makes sense. Thank you Richard forthe edits $\endgroup$
    – hela
    Commented Mar 27, 2022 at 17:19
  • $\begingroup$ @qwr because a statistical consultant told us to. We wanted to perform the linear regression but there were collinearities and almost non of the variables became significant. I assume when we want to do the regression it means we are doing an estimation but I am not sure what the prediction means and whether this has something to do with the test and train data. $\endgroup$
    – hela
    Commented Mar 27, 2022 at 17:22
  • $\begingroup$ If the model overall is significant, validated and well calibrated (e.g., you use the bootstrap to check validation and calibration), it doesn't matter if individual predictor coefficients aren't. You could try combining correlated predictors into single predictors to deal with collinearity; that might be preferable to lasso. See Harrell's course notes, in particular Section 4.7 on "Data Reduction. $\endgroup$
    – EdM
    Commented Mar 27, 2022 at 18:02

1 Answer 1

8
$\begingroup$

The penalization of coefficients with methods like lasso, adaptive lasso, and ridge regression means that you can model data even when the number of predictors exceeds the number of observations. You certainly have enough to use adaptive lasso, although this doesn't mean that the results will necessarily be as good as you might find with a larger data set.

If you had 100 times as many cases you might consider train/test splits. That only leads to trouble with data sets of this scale. You can validate your model-building process by repeating it on multiple bootstrap samples of your data and evaluating those models on the full data set.

Categorical predictors have to be handled carefully in penalized regressions, although there might be some simplification with adaptive lasso.

First, with standard lasso and ridge regression you want all predictors to be on comparable scales because you penalize all regression coefficients equally according to their magnitudes (lasso) or squared magnitudes (ridge). For continuous predictors that's accomplished via scaling to unit variance. But there's no single simple way to put categorical predictors into comparable scales versus each other or versus continuous predictors. The extra weighting of coefficient magnitudes in adaptive lasso inversely to initial estimate magnitudes might tend to minimize that problem.

Second, simple lasso by itself doesn't know that multiple regression coefficients correspond to the same multi-category predictor. You can specify that with the group lasso. So if you want all coefficients associated with a multi-category predictor to be retained or excluded together you would need to use an adaptive group lasso.

$\endgroup$
2
  • $\begingroup$ Million thanks EdM!! There is so much useful information that you gave on this answer that I have to go and train myself for Just a quick question. So overall you say there is no problem to use adaptive lasso but instead of having a train and test data, I can validate the model by running the lasso on multiple bootstraps (is it a simulation? Does that make sense when I am working with real data) because we have clinical data and so this is basically an analysis on 200 units who are patients with a specific health condition. Also, about group lasso, can I have a mix of continuous and categ? $\endgroup$
    – hela
    Commented Mar 27, 2022 at 17:31
  • $\begingroup$ @hela there's no simple, quick answer to your quick question. For the complexities, see Statistical Learning with Sparsity; Chapter 6 covers confidence intervals etc., which aren't straightforward with lasso. Combining continuous and categorical predictors in lasso is OK if you make proper decisions about scaling them. Bootstrapping simulates repeated sampling from the population. That's a standard validation method for clinical models, e.g. in the validate() function of the rms package. $\endgroup$
    – EdM
    Commented Mar 27, 2022 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.