This is a problem that I encountered in my experiment. I have 4 independent variables $x_1$, $x_2$, $x_3$, and $x_4$ and two dependent variables $y_1$ and $y_2$. My first trial is to regress $y_j=\sum_{i=1}^{4}{\beta_{i,j}\times x_i}, j=1, 2$ with two Lasso Regression. And later I compute $y=y_1+y_2$ knowing that this is a ground-truth formula with predicted $\hat{y}_1$ and $\hat{y}_2$. Then I tried to regress $y=\sum_{i=1}^{4}{w_i\times x_i}$ with one Lasso and do prediction $\hat{y}$. It turns out $\hat{y}=\hat{y}_1+\hat{y}_2$ and $w_i=\beta_{1,i}+\beta_{2,i}$ up to 3 decimal after the decimal point. One fact is that $w_i$ and $\beta_{i,j}$ are not large (at around 1 in absolute value).

If the model is a simple linear regression, then no problem I'm comfortable with $\hat{y}=\hat{y}_1+\hat{y}_2$. However, given the $L1$ regularization, will the equality definitely hold still? Or in this case, it happens to be simply because the weights are not large, thus the $L1$ regularization does not differ too much?

Any hints will be highly appreciated!

  • $\begingroup$ Actually, with Ridge Regression, i.e. the $L2$ regularization, the equality $\hat{y}=\hat{y}_1+\hat{y}_2$ still holds. $\endgroup$ – Summer_More_More_Tea Nov 17 '17 at 1:15
  • $\begingroup$ What is the benefit of adding $\hat{y}_1$ and $\hat{y}_2$ after estimation, rather than regressing $y = y_1 + y_2$ on the $x$s? $\endgroup$ – Frans Rodenburg Nov 17 '17 at 2:42
  • $\begingroup$ @FransRodenburg Just want to check which way is better, since I'm not sure whether the two ways are equivalent. $\endgroup$ – Summer_More_More_Tea Nov 17 '17 at 7:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.