I got the following question: Suppose I have three variables, $x_1$, $x_2$ and $y$. We run univariate regression of $y$ on $x_1$ ($x_2$) with intercept and get the regression coefficients $\beta_1$ ($\beta_2$). We then run a multivariate regression of $y$ on $x_1$ and $x_2$ and get the regression coefficients $\gamma_1$ and $\gamma_2$. We think about how $\gamma_1$ and $\gamma_2$ compared to $\beta_1$ and $\beta_2$ when there is no correlation between $x_1$ and $x_2$ and when $x_1$ and $x_2$ are highly correlated. In addition, if we run a Lasso model without normalizing the features, which one will have a smaller coefficient.
Here is what I attempted. First, if $x_1$ and $x_2$ are uncorrelated, then we have $\gamma_1 = \beta_1$ and $\gamma_2 = \beta_2$. The reason is that the variance-covariance matrix is diagonal. And we can show that the multivariate coefficients reduce to the simple univariate coefficients. When they are correlated, I was thinking about the partial coefficient formula
\begin{align*} \hat{\beta}_1 =\frac{\sqrt{var(x_1)var(y)}}{var(x_1)}\frac{r_{x_1y}-r_{x_2y}r_{x_1x_2}}{1-r_{x_1y}^2}. \end{align*}
However, it is unclear for me what happens when $r_{x_1, x_2}$ is close to one. In terms of the Lasso model, I was thinking if we do not apply any normalization so basically the regularization may be too large to the variable with smaller coefficient, relative to the other variable. So if the regularization is large enough, the one with smaller coefficient will be first shrunken to zero.
I am not sure whether my arguments are correct or not. In addition, I am also looking for some intuitive explanations of the questions.
