2
$\begingroup$

I am a beginner with LASSO. Is there any way (paper/code) to perform a LASSO type model in which only some of the explanatory variables are regularized? Or imposing different regulation parameters to several explanatory variables groups?

$\endgroup$
  • $\begingroup$ There's elastic net regularisation, which combines LASSO with Ridge. As for penalising only some of the variables couldn't you just use a weighted L1/L2-norm when calculating the penalty? As far as I can see the this should affect the computation procedure very little (i.e. you could use the same ones as for "regular" lasso/ridge/elastic net). That being said, I'm not aware of this being implemented anywhere. $\endgroup$ – Tilo Wiklund Jun 21 '13 at 14:14
3
$\begingroup$

I recommend using the glmnet package of Friedman and Tibshirani. The functions glmnet and cv.glmnet accept a parameter, 'penalty.factor', that appears to be what you need.

$\endgroup$
2
$\begingroup$

Yes, this is easily possible. If you use R, look into the penalized package.

$\endgroup$
1
$\begingroup$

You can do it by hand with the use of nlm in R. The caveat being that an "all purpose" optimizer such as nlm can converge to local minima or diverge and is particularly slow. Contrast with the specialized gradient descent code that Tibshirani and those guys made to estimate LASSO models: it is much more regular and ridiculously fast. Obviously, this introduces some sample size issues. With unconstrained OLS, you necessarily have to have $p \ll n$, LASSO lets you have $p \gg n$. This maximum likelihood, err maximum penalized likelihood, err maximum product likelihood penalized likelihood approach is constrained in the OLS way.

  1. calculate residuals' log product density in the normal probability model
  2. subtract penalized parameter and L1 norm for LASSO'd parameters
  3. maximize partially penalized likelihood in nlm.

Example below:

set.seed(4)
n <- 100
p <- 10
x <- matrix(rnorm(n*p), n, p)
X <- model.matrix(~x)
y <- rnorm(n)


## i: index of which parameters aren't tuned
## l: tuning parameter
LASSO.OLS.LOVECHILD <- function(i, l) {
  penLogLik <- function(b) {
    -sum(dnorm(y - b%*%t(X), log=TRUE)) + l*sum(abs(b[!i]))
  }
  b <- vector('numeric', p+1)
  val <- nlm(penLogLik, b)
  val$estimate
}

## ordinary least squares 1 way: all unpenalized betas
LASSO.OLS.LOVECHILD(i=rep(TRUE, p), l=20)
lm(y ~ x)

## ordinary least squares another way (no tuning)
LASSO.OLS.LOVECHILD(i=rep(FALSE, p), l=0)
lm(y ~ x)

## unconstrained first half, heavy tuning forces last 5 to 0
## (not too heavy or else algorithm doesn't converge to anything...)
round(LASSO.OLS.LOVECHILD(i=1:10 %in% 1:5, l=20), 2)
round(lm(y ~ x[, 1:5])$coef, 2)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.