Skip to main content
edited body
Source Link
skijunkie
  • 511
  • 4
  • 7

$$ X^tX\beta-\frac{X^ty}{s_y}+N\lambda\beta =0 $$$$ X^TX\beta-\frac{X^Ty}{s_y}+N\lambda\beta =0 $$

$$ \tilde{\beta}_{GLMNET}= (X^tX+N\lambda I_p)^{-1}\frac{X^ty}{s_y} $$$$ \tilde{\beta}_{GLMNET}= (X^TX+N\lambda I_p)^{-1}\frac{X^Ty}{s_y} $$

$$ \hat{\beta}_{GLMNET}=s_y\tilde{\beta}_{GLMNET}= (X^tX+N\lambda I_p)^{-1}X^ty $$$$ \hat{\beta}_{GLMNET}=s_y\tilde{\beta}_{GLMNET}= (X^TX+N\lambda I_p)^{-1}X^Ty $$ $$ \lambda_{unstd.}=s_y\lambda $$

$$ \hat{\beta}= (X^tX+\lambda I_p)^{-1}X^ty $$$$ \hat{\beta}= (X^TX+\lambda I_p)^{-1}X^Ty $$

$$ X^tX\beta-\frac{X^ty}{s_y}+N\lambda\beta =0 $$

$$ \tilde{\beta}_{GLMNET}= (X^tX+N\lambda I_p)^{-1}\frac{X^ty}{s_y} $$

$$ \hat{\beta}_{GLMNET}=s_y\tilde{\beta}_{GLMNET}= (X^tX+N\lambda I_p)^{-1}X^ty $$ $$ \lambda_{unstd.}=s_y\lambda $$

$$ \hat{\beta}= (X^tX+\lambda I_p)^{-1}X^ty $$

$$ X^TX\beta-\frac{X^Ty}{s_y}+N\lambda\beta =0 $$

$$ \tilde{\beta}_{GLMNET}= (X^TX+N\lambda I_p)^{-1}\frac{X^Ty}{s_y} $$

$$ \hat{\beta}_{GLMNET}=s_y\tilde{\beta}_{GLMNET}= (X^TX+N\lambda I_p)^{-1}X^Ty $$ $$ \lambda_{unstd.}=s_y\lambda $$

$$ \hat{\beta}= (X^TX+\lambda I_p)^{-1}X^Ty $$

Added example for standardized X with no intercept, fixed notation mistakes
Source Link
skijunkie
  • 511
  • 4
  • 7

Added code to show standardized X with no intercept:

set.seed(123)

n <- 1000
p <-  100
X <- matrix(rnorm(n*p,0,1),n,p)
beta <- rnorm(p,0,1)
Y <- X%*%beta+rnorm(n,0,0.5)

sd_y <- sqrt(var(Y)*(n-1)/n)[1,1]

mean_x <- colMeans(X)
sd_x <- sqrt(apply(X,2,var)*(n-1)/n)

X_scaled <- matrix(NA, nrow = n, ncol = p)
for(i in 1:p){
    X_scaled[,i] <- (X[,i] - mean_x[i])/sd_x[i] 
}

beta1 <- solve(t(X_scaled)%*%X_scaled+10*diag(p),t(X_scaled)%*%(Y))[,1]

fit_glmnet <- glmnet(X_scaled,Y, alpha=0, standardize = F, intercept = 
FALSE, thresh = 1e-20)
beta2 <- as.vector(coef(fit_glmnet, s = sd_y*10/n, exact = TRUE))[-1]
cbind(beta1[1:10], beta2[1:10])

             [,1]        [,2]
 [1,]  0.23560948  0.23560948
 [2,]  1.83469846  1.83469846
 [3,] -0.05827086 -0.05827086
 [4,] -0.04927314 -0.04927314
 [5,]  0.41871870  0.41871870
 [6,]  1.28969361  1.28969361
 [7,]  0.06552927  0.06552927
 [8,]  0.44576008  0.44576008
 [9,]  0.90156795  0.90156795
[10,] -2.43163420 -2.43163420

Added code to show standardized X with no intercept:

set.seed(123)

n <- 1000
p <-  100
X <- matrix(rnorm(n*p,0,1),n,p)
beta <- rnorm(p,0,1)
Y <- X%*%beta+rnorm(n,0,0.5)

sd_y <- sqrt(var(Y)*(n-1)/n)[1,1]

mean_x <- colMeans(X)
sd_x <- sqrt(apply(X,2,var)*(n-1)/n)

X_scaled <- matrix(NA, nrow = n, ncol = p)
for(i in 1:p){
    X_scaled[,i] <- (X[,i] - mean_x[i])/sd_x[i] 
}

beta1 <- solve(t(X_scaled)%*%X_scaled+10*diag(p),t(X_scaled)%*%(Y))[,1]

fit_glmnet <- glmnet(X_scaled,Y, alpha=0, standardize = F, intercept = 
FALSE, thresh = 1e-20)
beta2 <- as.vector(coef(fit_glmnet, s = sd_y*10/n, exact = TRUE))[-1]
cbind(beta1[1:10], beta2[1:10])

             [,1]        [,2]
 [1,]  0.23560948  0.23560948
 [2,]  1.83469846  1.83469846
 [3,] -0.05827086 -0.05827086
 [4,] -0.04927314 -0.04927314
 [5,]  0.41871870  0.41871870
 [6,]  1.28969361  1.28969361
 [7,]  0.06552927  0.06552927
 [8,]  0.44576008  0.44576008
 [9,]  0.90156795  0.90156795
[10,] -2.43163420 -2.43163420
added 6 characters in body
Source Link
skijunkie
  • 511
  • 4
  • 7

$$ \tilde{\beta}_{GLMNET}= (X^tX+N\lambda I)^{-1}\frac{X^ty}{s_y} $$$$ \tilde{\beta}_{GLMNET}= (X^tX+N\lambda I_p)^{-1}\frac{X^ty}{s_y} $$

$$ \hat{\beta}_{GLMNET}=s_y\tilde{\beta}_{GLMNET}= (X^tX+N\lambda I)^{-1}X^ty $$$$ \hat{\beta}_{GLMNET}=s_y\tilde{\beta}_{GLMNET}= (X^tX+N\lambda I_p)^{-1}X^ty $$ $$ \lambda_{unstd.}=s_y\lambda $$

$$ \hat{\beta}= (X^tX+\lambda I)^{-1}X^ty $$$$ \hat{\beta}= (X^tX+\lambda I_p)^{-1}X^ty $$

$$ \tilde{\beta}_{GLMNET}= (X^tX+N\lambda I)^{-1}\frac{X^ty}{s_y} $$

$$ \hat{\beta}_{GLMNET}=s_y\tilde{\beta}_{GLMNET}= (X^tX+N\lambda I)^{-1}X^ty $$ $$ \lambda_{unstd.}=s_y\lambda $$

$$ \hat{\beta}= (X^tX+\lambda I)^{-1}X^ty $$

$$ \tilde{\beta}_{GLMNET}= (X^tX+N\lambda I_p)^{-1}\frac{X^ty}{s_y} $$

$$ \hat{\beta}_{GLMNET}=s_y\tilde{\beta}_{GLMNET}= (X^tX+N\lambda I_p)^{-1}X^ty $$ $$ \lambda_{unstd.}=s_y\lambda $$

$$ \hat{\beta}= (X^tX+\lambda I_p)^{-1}X^ty $$

deleted 4 characters in body
Source Link
skijunkie
  • 511
  • 4
  • 7
Loading
Bounty Ended with 50 reputation awarded by amoeba
Source Link
skijunkie
  • 511
  • 4
  • 7
Loading