0
$\begingroup$

I understand that, with LASSO, the regularization term puts a constraint on the complexity of our regression model. Usually, for prediction applications, regularization makes the model perform better in out of sample data set.

What I cannot understand is that, why regularization term could help in a multi-response setup.

For instance, we have 2 predictors, $f_{1}$ and $f_{2}$, and we have 2 responses, $y_{1}$ and $y_{2}$, we are searching for a 2 x 2 matrix of betas, where $\beta_{1}^{1}$ and $\beta_{1}^{2}$ represent the coefficient of $f_{1}$ for the first and second responses.

Simply speaking, there are two scenarios, first: the responses are correlated, then we expect that the coefficients $\beta_{1}^{1}$ and $\beta_{1}^{2}$ should be similar. The second scenario, responses are not correlated or negatively correlated, $\beta_{1}^{1}$ and $\beta_{1}^{2}$ should be distinct.

What I understand after reading glmnet doct is that, through the regularization term, we enforce that the norm of vector $[\beta_{1}^{1}, \beta_{1}^{2}]$ is small. However, this does NOT enforce any relationship as we described above, right?

Therefore, my question is how could such 'joint' regularization constraint better than 2 individual LASSO fits? or in other words, what could go wrong if we run 2 LASSO fits for $y_{1}$ and $y_{2}$ individually?

Just to make sure we are all talking about the same thing: in the linked document, the penalty term is

$(1-\alpha)||\beta||_{F}^{2}/2 + \alpha\sum_{j=1}^{p}||\beta_{j}||_{2}$

using GLMNET's terminology , when $\alpha = 1$, we have LASSO

so the penalty term is $\alpha\sum_{j=1}^{p}||\beta_{j}||_{2}$, which is the collection of $L2$ normal of each row of betas, i.e., betas for each response for a specific predictor

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.