3
$\begingroup$

I have a data regarding life satisfaction, of more than 2000 observations and 265 variables (most are categorical variables). I want to build a model, estimating the effects of society problems on the life satisfaction in the USA.

So firstly, I use lasso2 to choose the variables which are most suitable for the regression, as well as reduce the dimension. It then results in 98 selected variables, however this is still a very large number of variables for a regression. Therefore, should I use PCA or Factor Analysis to combine these 98 variables into some factors, and then use these factors to regress?

Is it possible anyway? I am afraid that, PCA/ FA and LASSO do the same thing, which is dimension reduction, and we can not use them at the same time. If so, which methods should I use after LASSO, in order to create factors, and use the factors to build a model?

Thank you very much!

Improve this question
$\endgroup$
9
  • $\begingroup$ LASSO has a parameter $\lambda$ which determines how much you regularize your model, setting his value higher will incur a higher penalty and the model will select fewer variables. $\endgroup$ – user2974951 Jan 10 '19 at 8:26
  • $\begingroup$ thank you very much for your help! However, I am still not clear about how to choose the lambda value, what is the optimal value I should choose? or is there any maximum value of lambda? $\endgroup$ – Thi Van Anh Pham Jan 10 '19 at 12:58
  • $\begingroup$ And my aim is to consider the effects of society on the happiness, but society is not already a variable. Therefore, I want to use PCA or FA to combine many highly correlated variables to different factors (including society factors). Is it still possible? Or somehow the two methods (lasso and PCA) duplicates? $\endgroup$ – Thi Van Anh Pham Jan 10 '19 at 13:01
  • $\begingroup$ $\lambda$ is usually chosen based on CV, this should be automatic. PCA and LASSO are different, PCA does dimensionality reduction, LASSO does feature selection. You need to decide what you want. Also by society is not already a variable I think you mean that society is a latent variable, to find common factors you would use FA. $\endgroup$ – user2974951 Jan 10 '19 at 13:41
  • 1
    $\begingroup$ It sounds like you don't need LASSO at all. Why not just do PCA directly and fit the model on the PCA output? $\endgroup$ – shadowtalker Jan 11 '19 at 0:08
2
$\begingroup$

PCA and LASSO are two very different approaches and usually employed to very different ends.

Lasso regression is not a dimension reduction technique but can be used for variable selection. Typically, when people use lasso (or other regularization techniques) the goal of the exercise is to attain a model which is good for prediction, although it also will tell you which variables are related to your response, how, and to what degree. The regularization that ridge employs helps account for multi-collinearity, and restricts the coefficients so they don't fit noise in your data, and (ideally) only fits the true signal produced by those effects.

Principle components analysis (PCA) on the other hand is a dimension reduction technique, meaning that it takes n dimensions in the data, identifies the orthogonal axes of variation in the n-dimensional cloud, starting with the axis of greatest variation, and continues in the same manner until it has n axes. In this case there is no response variable and these models are not typically used for prediction, so much as understanding the relationships between all the variables simultaneously. There are multivariate techniques that can be used for regression though, such as linear discriminant analysis (LDA).

What is the goal of your study? Is it purely prediction? Are you trying to understand the underlying relationships between variables? Some combination of both? If it is the latter then I would suggest you build separate models as predictive ones are not often easily interpreted when you have so many factors. This is because techniques that are good for prediction usually involve variable selection which is heavily frowned upon for purposes other than prediction as it promotes 'fishing expeditions' as you hunt for 'significant' relationships. In contrast, techniques which do not have a response variable (i.e. PCA) are used for hypothesis generation and do not rely on 'significance'.

If you clarify your goals more I am happy to offer advice on how to proceed.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I am working on my thesis with the topic " Factors determining life satisfaction in the USA", and have a data set with 265 variables. I choose happiness as the response variables. My professor told me to fist choose the most suitable variables for regression by LASSO. And he agreed with the 98 selected variables. Then he wants me to, either use some factors to regress, or choose just a few variables to regress. I want to follow the 1st direction, but I only know PCA or FA to call out the latent variables, and combine them to some factors. Could you please recommend me a more effective method? $\endgroup$ – Thi Van Anh Pham Jan 11 '19 at 15:01
  • $\begingroup$ I would personally start with just a PCA on the data, check the scree plot/eigenvalues, and look at a plot of the PCAs first two axes. I am afraid I can't recommend much past that without seeing those outputs. $\endgroup$ – André.B Jan 14 '19 at 20:37
  • $\begingroup$ Using LASSO blindly (or any other method) is potentially very dangerous. The reason is that LASSO would tend to kick out covariates with small coefficients even though they are 'good' predictors. Variable selection is a tricky issue. Here is one research paper that might help to understand why this is the case and what could be done: faculty.chicagobooth.edu/christian.hansen/research/…. They provide detailed discussion and example codes (with STATA and R). $\endgroup$ – Jonas Striaukas Mar 19 '20 at 13:26
  • $\begingroup$ Why not trying clustering covariates into groups, taking PCA of each group, and then run LASSO on those. You can also have large number of factors per cluster and allow LASSO to select; within-group factors would be orthogonal and this surely helps in selection problems. This would sound like a more valid approach. $\endgroup$ – Jonas Striaukas Mar 19 '20 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.