New answers tagged

1

Typically, $X$ is $n\times p$, where $n$ is the number of samples and $p$ is the number of features. So, the shape of $\lambda I$ should be $p\times p$ in order for the expression to be mathematically correct. The identity term comes from the loss function of the Ridge Regression: $$L=||X\beta-y||^2+{\lambda}||\beta||^2 = (X\beta-y)^T(X\beta-y)+{\lambda}\...


0

No @whuber was not really recommending this approach. I would say the standard software approach, at least in R, is using offset. Ie if you want to regularise around b instead of 0, then add an offset of Xb, then your coefficients are for (Beta-b) See offset Param in https://www.rdocumentation.org/packages/lme4/versions/1.1-21/topics/lmer


1

If you use no regularization, i.e. alpha=0, they'll be the same/very similar when used the same random initialization, e.g. random_state=42. This is because Ridge class doesn't regularize the intercept term, but only the weights. When you include the intercept term, it becomes a member of your feature set and the column is assigned a weight, which is then ...


0

It says that $U \in \mathbb{R}^{n \times p}$ so the columns are orthogonal but the matrix is not necessarily orthogonal in the sense that $UU^T =I$ ($\dim(UU^T) =p$)


2

$U$ has orthogonal columns but it's not square and isn't an orthogonal matrix. $UU^T \neq I$ in general since it only has rank $p$ and $UU^T$ involves dot products of the rows which don't have to be orthogonal when $p<n$. If $U$ was square then it would be the case that $UU^T = I$ which makes sense since then $\hat y = y$ since $X$ is a bijection. I'm ...


4

From the context, I'm assuming that the $\beta_j's$ are the regular least squares estimates, and the table is showing how they would be transformed under each of the listed methods. Best Subset: Because the columns are orthonormal, the least squares coefficients are simply $\hat{B_j} = {x_j^{T}y}$. (Orthogonality implies that they're given by $\hat{B_j} = ...


1

Yes, you can obtain a rough estimate of the penalty parameter from standard OLS output. People in the old days used to compute $$ \lambda = \frac{ k \hat{\sigma }^2} {||\hat{\boldsymbol{\beta}}||^2}, $$ where $k$ is the number of slope coefficients, $ \hat{\sigma }$ is an estimated residual scale and $\hat{\boldsymbol{\beta}} $ is the vector of estimated ...


2

It looks like you might be confusing the Penalized and Lagrangian forms of ridge regression, and trying to do both at once. See e.g. One-to-one correspondence between penalty parameters of equivalent formulations of penalised regression methods


Top 50 recent answers are included