New answers tagged

1

That's right. In the case of ordinary lieast squares (OLS), $Y - \hat{Y} = (I-X(X'X)^{-1}X')Y$ and the fact that $$(I-X(X'X)^{-1}X')X = 0$$ implies that $Y - \hat{Y}$ is orthogonal to any linear combination of the $X$ (and hence to $\hat{Y} = X\hat{\beta}$). However, it is no longer true that $$(I-X(X'X+ \lambda I)^{-1}X')X = 0$$ so orthogonality to $\hat{Y}...


1

This is due to the fact that you are using L2 regularization on a particular model setup where the coefficients represent some sort of difference between group 1 and a reference group 0. If you set up the coefficients differently (e.g., with respect to a different reference group, or e.g. removing the intercept and encoding the group means directly as ...


3

I don't think you really need to calculate the inverse for the purpose. Substitute $\nabla L$ to the update function, you get $$ \begin{align} \theta^{(t+1)} &= \theta^{(t)} - H^{-1} \left[ (X^t X + \lambda I_n) \theta^{(t)} - X^ty\right] \\ &= \theta^{(t)} - H^{-1} H \theta^{(t)} + H^{-1} X^ty \\ &= H^{-1} X^ty \end{align} $$ This is the closed ...


0

The answer is it depends on what you try to achieve. Post-LASSO OLS estimator is well understood, see e.g. [1]. What has to be said is that you can never be sure that the selected covariates are 'true' covariates - LASSO, like any other estimator (e.g. adaptive LASSO) does mistakes. Oracle properties of adaptive LASSO are derived under very strong ...


1

The usual practice in penalized methods like LASSO, ridge or principal-component regression is to start by standardizing all predictors to zero mean and unit standard deviation. Otherwise the issue that you raise would lead to a distance measured in miles being handled differently than the same distance measured in millimeters. That's the default, for ...


0

Who is your professor? The model he has assigned comes from the following paper: de Miguel et al (2009) A Generalized Approach to Portfolio Optimization: Improving Performance by Constraining Portfolio Norms Instead of using an additive penalty term, the ridge shrinkage of the portfolio weight vector should, or works best, as a separate constraint: $${\...


4

The problem is that ridge regression and lasso and the like are all used to attend model selection tasks and are not necessarily coded for a univariable use-case. If you want to do a univariable regression, you could try to use parcor::lm.ridge.univariate. If you attempt to do univariable ridge regression (predictor matrix X is a vector) with the function ...


5

If $\lambda$ is a nonzero eigenvalue of an invertible matrix $A$, then $\lambda^{-1}$ is an eigenvalue of $A^{-1}$. To show this, note that if $Av=\lambda v$ then $v = A^{-1}(\lambda v) = \lambda A^{-1} v$ and thus $\lambda^{-1} v = A^{-1} v$.


Top 50 recent answers are included