8
$\begingroup$

I have a mixed model with many fixed effects, and two random variables (the data is made up of repeat observations of a number of individuals, and has a random intercept and a random slope parameter associated with each individual). The model will be used for prediction. I want to use a regularisation technique to reduce the number of fixed effects as the model is likely overfitted at present.

Would it be valid to run a Lasso analysis on a model which excludes the random effects and use the outcome of this to choose the fixed effects to keep in my mixed model? As in:

  1. Run a number of Lasso regressions on a training set where the tuning parameter (lambda) is varied
  2. Calculate the RMSE of an (unseen) test set
  3. Find the value of lambda which gives the best RMSE in the test set
  4. Find the variables for which the coefficient is set to zero for the value of lambda that produces the best RMSE in the test set
  5. Exclude these variables from a subsequent mixed model.

I've found that there are lots of papers for different lasso-type things that people do for mixed models, but I'm struggling to find a clear, reliable, reference that explains why you couldn't just do the steps above... My intuition is that this would not be valid because the Lasso wouldn't know about the structure in the model that the random effects capture... is that right? And if so, could you point me to a nice reference that makes this clear? Or, if it is ok, could anyone point me to a nice reference that makes that clear?

I hope this makes sense, and sorry if it's a simple answer!

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Do you plan to do any kind of inference once you have your mixed model? Hypothesis testing? Confidence intervals? Coefficient point estimates? $\endgroup$
    – Dave
    Commented Jun 25, 2021 at 17:22
  • $\begingroup$ One consideration: Singer & Willett make clear that OLS regression using clustered data will produce unbiased fixed-effect coefficients, even though standard errors will be too small. amazon.com/Applied-Longitudinal-Data-Analysis-Occurrence/dp/… $\endgroup$
    – rolando2
    Commented Jun 25, 2021 at 18:39
  • $\begingroup$ ...Singer & Willett, p. 86. $\endgroup$
    – rolando2
    Commented Jun 25, 2021 at 20:27
  • $\begingroup$ @Dave I'm planning to use the model for prediction, I'll edit the OP to make that clear. $\endgroup$
    – skittle
    Commented Jun 29, 2021 at 12:39
  • $\begingroup$ @rolando2 thanks, that's useful, but would it stand for Lasso regression as well though? I think since I have multiple observations for participants the residuals won't be independent... I'm not sure if that would affect Lasso in a different way. $\endgroup$
    – skittle
    Commented Jun 29, 2021 at 12:43

2 Answers 2

Reset to default
6
$\begingroup$

There doesn't appear to be a consensus on how to perform variable selection on both fixed and random effects. There are technical papers proposing solutions to this problem, like this paper from Fan and Li.

Bondell et al. argue against separating the fixed and random when performing variable selection, as the structure of the random effects will affect which fixed effect variables are selected. I am not an expert at variable selection or mixed models, but Bondell's claims and your intuition seem correct to me. Since this problem doesn't seem to be fully solved, I don't think you'll find the clear resource you want. You will find many technical arguments like those papers or their references.

It is straightforward to produce an example where temporarily ignoring the random effects will not work. Imagine you have a longitudinal study and a random effect for each person. It wouldn't make any sense to omit the random effects because that is how you model within-person correlation.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ (+1) nice answer ! $\endgroup$
    – Robert Long
    Commented Jun 25, 2021 at 19:31
  • $\begingroup$ Does my comment above change your opinion in your 3rd paragraph? $\endgroup$
    – rolando2
    Commented Jun 25, 2021 at 20:23
1
$\begingroup$

From a machine learning standpoint, you may not touch your test set until the last step of your analyses where you measure your performance. Your two-phased training is a clear example of information leakage.

I believe you can combine both models into a single framework where you "ignore" your random effects in the regularization part of your loss function. This is already supported in glmnet.

Note: Posted links are not necessarily the best reference. They are simply my first Google hits.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Well said. Using lasso as a screening device will ignore the shrinkage lasso rightly provides. But beware that lasso is not reliable in finding the "right" features in many cases anyway. Best to put the whole process into a single Bayesian model with random effects and better shrinkage priors than the Laplace approach used by lasso. $\endgroup$
    – Frank Harrell
    Commented Nov 16, 2023 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.