# Tag Info

That's right. In the case of ordinary lieast squares (OLS), $Y - \hat{Y} = (I-X(X'X)^{-1}X')Y$ and the fact that $$(I-X(X'X)^{-1}X')X = 0$$ implies that $Y - \hat{Y}$ is orthogonal to any linear combination of the $X$ (and hence to $\hat{Y} = X\hat{\beta}$). However, it is no longer true that $$(I-X(X'X+ \lambda I)^{-1}X')X = 0$$ so orthogonality to $\hat{Y}... 1 This is due to the fact that you are using L2 regularization on a particular model setup where the coefficients represent some sort of difference between group 1 and a reference group 0. If you set up the coefficients differently (e.g., with respect to a different reference group, or e.g. removing the intercept and encoding the group means directly as ... 3 I don't think you really need to calculate the inverse for the purpose. Substitute$\nabla Lto the update function, you get \begin{align} \theta^{(t+1)} &= \theta^{(t)} - H^{-1} \left[ (X^t X + \lambda I_n) \theta^{(t)} - X^ty\right] \\ &= \theta^{(t)} - H^{-1} H \theta^{(t)} + H^{-1} X^ty \\ &= H^{-1} X^ty \end{align} This is the closed ... 0 The answer is it depends on what you try to achieve. Post-LASSO OLS estimator is well understood, see e.g. [1]. What has to be said is that you can never be sure that the selected covariates are 'true' covariates - LASSO, like any other estimator (e.g. adaptive LASSO) does mistakes. Oracle properties of adaptive LASSO are derived under very strong ... 1 The usual practice in penalized methods like LASSO, ridge or principal-component regression is to start by standardizing all predictors to zero mean and unit standard deviation. Otherwise the issue that you raise would lead to a distance measured in miles being handled differently than the same distance measured in millimeters. That's the default, for ... 0 Who is your professor? The model he has assigned comes from the following paper: de Miguel et al (2009) A Generalized Approach to Portfolio Optimization: Improving Performance by Constraining Portfolio Norms Instead of using an additive penalty term, the ridge shrinkage of the portfolio weight vector should, or works best, as a separate constraint:${\... 4 The problem is that ridge regression and lasso and the like are all used to attend model selection tasks and are not necessarily coded for a univariable use-case. If you want to do a univariable regression, you could try to use parcor::lm.ridge.univariate. If you attempt to do univariable ridge regression (predictor matrix X is a vector) with the function ... 5 If$\lambda$is a nonzero eigenvalue of an invertible matrix$A$, then$\lambda^{-1}$is an eigenvalue of$A^{-1}$. To show this, note that if$Av=\lambda v$then$v = A^{-1}(\lambda v) = \lambda A^{-1} v$and thus$\lambda^{-1} v = A^{-1} v\$.