let say I have M subsets of independent variables and I want to use stack learner to predict dependent variable y.
for each subset I use lasso method to get meta features (predictions).

I have 2 options and Im not sure which one is better:

  1. try alot of $\lambda s$ for each subset and stack all the models so I get (number of $\lambda$s tested * num. of subsets) models for stacking.
  2. find the best $\lambda$ for each subset and stack only M models.

I suspect option number one will lead to overfit but maybe can do better performances.

which of the options is better for generalize?

thanks ahead

  • $\begingroup$ (For Opt. 1): We will probably test dozens of $\lambda$'s, some of them will be rubbish, some will not be. There is no reason to feed a world of junk in our stack-learner. (Generally:) That said if you use LASSO as the stack-lerner well as for the base-learner you essentially take linear combinations of linear combinations, which are once again linear combinations. I strongly suspect that the end model will be at best equivalent to a LASSO with differential shrinkage (so the amount of regularisation $\lambda$ could vary between covariates). I would suggest you use a variety of base-learners. $\endgroup$ – usεr11852 Dec 9 '17 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.