2
$\begingroup$

I'm a complete newbie and trying to understand what exactly LASSO is, how coordinate descent is used with LASSO, and how all of that factors into regression analysis. I'm totally confused about the interplay between these three things.

The key thing I'm really trying to clarify is terminology - from my reading, it sounds like LASSO is a "technique" or "method" for determining "regression coefficients" for generating a "model" of data. But LASSO has to be "solved" to find the regression coefficients, and one way to solve LASSO is to use a coordinate descent "technique" or "method". As I'm sure you can tell by now, I'm not clear on what output coordinate descent gives you, how that output is used in LASSO, and how LASSO is then used to perform regression analysis to generate a model of data.

Assume I have literally zero statistics training and this needs to be explained to a five year old. Any and all clarification would be greatly appreciated.

$\endgroup$
  • $\begingroup$ For ordinary least squares regression (OLS) the loss function that we try to minimize is $\min_{\beta \in \Omega} (y-x\beta)^2$. In lasso regression the loss function that we try to minimize is $min_{\beta \in \Omega} (y-x\beta)^2+\lambda\|\beta\|$. The nature of the loss function in lasso regression requires us to use optimization algorithms and one such algorithm that is widely used is a coordinate descent method. That's my understanding of the topic. $\endgroup$ – A Gore Apr 20 '16 at 23:12
  • $\begingroup$ Thanks for your answer! But this is a bit too technical for me. Can you explain what some of the variables in these equations stand for in English, and what you mean by "optimization algorithm" (what exactly are we optimizing)? $\endgroup$ – icydash Apr 21 '16 at 13:02
2
$\begingroup$

Regression analysis is a statistical process for estimating the relationships among variables. For example, estimating the relationship the weather (input variable) has on stock market (output variable). Multivariate regression is when you want to estimate the effects of several input variables on one output variable. For example lets say you simultaneously wanted to estimate the effects that the weather, national unemployment, interest rate, and price of milk all have on the stock market.

LASSO regression is used when you are doing a multivariate regression problem and you want to know which of your input variables have an actual effect on the output variable - i.e. you want to estimate a sparse model. The advantage of LASSO over normal regression is that it will theoretically set the coefficients of all "non-relevant" variables to 0. In ordinary regression, due to noise and limited data, even if 1 of the input variables is "non-relevant", the coefficient assigned to it will probably be small, but not not 0.

In our above example, lets say weather had no effect on stock prices. In LASSO regression, its coefficient will be 0, while in normal regression you might get something like .0123.

This may not sound like a big difference, but in the age of big data, where we can have problems with 1000s of input variables, LASSO regression becomes extremely useful to see which inputs are casually related to the output.

This brings us to coordinate descent. LASSO regression just means finding the coefficients to minimize the LASSO cost function. This requires a specific algorithm/method. The method used to do this is coordinate descent. This is how the behind the curtain magic of LASSO happens. Note, there is nothing magical about this method: you could in principle get the same estimates by trying all combinations of numbers and just choosing the ones which give you the lowest cost function. Coordinate descent just accomplishes the same thing in a quick and accurate manner. If you are just using a statistical package to get the LASSO estimates, this isnt so important for you since the package implements this algorithm for you.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks!! This definitely helped a lot. But I think I'm still unclear on terminology. Say you have a data set. Would it be correct to say that you "determine a model" of the data in the data set using a "LASSO regression method" by minimizing the "LASSO cost function" using a "coordinate descent technique"? If I understand it, coordinate descent gives you a number that minimizes the LASSO cost function, and you plug that number into the LASSO equation to determine a coefficient for a particular variable, and all the coefficients for all the variables make up your "model" of the data? $\endgroup$ – icydash Apr 21 '16 at 13:13
  • $\begingroup$ Sort of, but not quite. Coordinate descent gives you a parameter vector of all your model coefficients. These coefficients are optimal in the sense that, if they were plugged into the LASSO cost function they would minimize it. No other vector of coefficients would give a lower value. $\endgroup$ – DankMasterDan Apr 22 '16 at 3:37
  • $\begingroup$ Ok, so you actually get all the coefficients for the model (in a single vector) at the output of coordinate descent? And, as part of the coordinate descent algorithm, what it's trying to do is determine the coefficients that would minimize the LASSO cost function if plugged into the LASSO cost function? Also, does the term "coordinate descent" imply LASSO? In other words, is coordinate descent ONLY used to minimize the LASSO cost function? $\endgroup$ – icydash Apr 22 '16 at 19:08
  • $\begingroup$ the description sounds correct. Coordinate descent to also used in other algorithms, and is not exclusive to LASSO: en.wikipedia.org/wiki/Coordinate_descent#Applications $\endgroup$ – DankMasterDan Apr 22 '16 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.