6
$\begingroup$

As you know lasso is a popular variable selection method of the form of

$ (y-x\beta)'(y-X\beta)+\lambda \sum_i|\beta_i| $

the first is that it is possible to use optim() function in R to minimize this problem? a sample code can be like

x=matrix(rnorm(100),ncol=20)
y=rowSums(x)
f<-function(x,y,l,beta){
  beta=as.matrix(beta)
  sum((y-x%*% beta)^2) +l*sum(abs(beta))
}
optim(rep(0,ncol(x)),f,method='CG',x=x,y=y,l=1)

Other questions are, 2) is the code above true? 3) how can I force the coefficients to be exactly zero?

PLEASE NOTICE THAT I DONT WANT TO USE PACKAGES LIKE LARS, GLMNET or ... just optim or nlm functions. Thanks

$\endgroup$
8
  • 1
    $\begingroup$ is this self study? $\endgroup$
    – user603
    Oct 23, 2014 at 19:00
  • $\begingroup$ By construction, your coefficients will be estimated near 1. To drop the smallest coefficient(s), you will need to experiment with increasing the value of l. But it might be more illustrative to make one of the coefficients smaller than the others so that it is consistently dropped. $\endgroup$
    – Sycorax
    Oct 23, 2014 at 19:28
  • $\begingroup$ @user777 Thank you. why 1? can you explain that more? $\endgroup$
    – TPArrow
    Oct 23, 2014 at 20:06
  • $\begingroup$ You created y by summing over x. $\endgroup$
    – Sycorax
    Oct 23, 2014 at 20:07
  • $\begingroup$ Sorry, Yes it is. But if generate a sparse matrix and execute the code above you will see that the zero coefficients are not zero. My main challenge is tackling with this coefficients. Maybe I need to change the input data above to sparse data. $\endgroup$
    – TPArrow
    Oct 23, 2014 at 20:11

1 Answer 1

5
$\begingroup$

With standard algorithms for convex smooth optimization, like CG, gradient descent, etc, you tend to get results that are similar to lasso but the coefficients don't become exactly zero. The function being minimized isn't differentiable at zero so unless you hit zero exactly, you're likely to get all coefficients non-zero (but some very small, depending on your step size). That's why lasso and similar specialized algorithms are useful.

But if you insist on using these algorithms, you can truncate values, e.g., once you've got the "optimal" solution set all betas under 1e-9 or something to zero.

$\endgroup$
2
  • $\begingroup$ What about a an algorithm that does not assume smoothness, eg, Nelder-Mead? (Not that I would actually recommend that except in toy examples). $\endgroup$
    – Andrew M
    Oct 24, 2014 at 6:46
  • $\begingroup$ From my (limited) experience these types of algorithms tend to bounce around a lot, and don't seem to give satisfactory results compared to proximal methods (soft thresholding via gradient or coordinate descent), relative to the amount of effort. But why do it the hard way when there are good methods? $\endgroup$
    – purple51
    Oct 24, 2014 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.