I have about 40 candidate dichotomous predictors. I want to know which ones predict a DV, and in what way. Is an adaptive LASSO regression a good way to do this?
If not, could you explain why not, and recommend something better?
$\begingroup$
$\endgroup$
4
-
1$\begingroup$ Does this answer your question? Lasso vs. adaptive Lasso $\endgroup$– develaristAug 28, 2020 at 1:59
-
4$\begingroup$ The problem is much bigger than LASSO vs adaptive LASSO. $\endgroup$– kurtosisAug 28, 2020 at 2:30
-
$\begingroup$ While I'm no statistician myself; are you perhaps looking for the broader concept of Dimentionality Reduction using techniques such as PCA (Principle Component Analysis)? $\endgroup$– Cameron KerrSep 9, 2020 at 8:13
-
$\begingroup$ @CameronKerr Model selection, not dimensionality reduction, since prediction is the idea. Dimensionality reduction might help... or might not, since it is done without regard to the response. $\endgroup$– kurtosisSep 12, 2020 at 19:18
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
As a general rule, regression models with penalties are reasonably good at variable selection. (Better than the bad old days of stepwise procedures anyway!) Penalty models usually have some consistency properties that ensure accurate selection of variables for large samples under certain conditions on the penalties. The goal of these models is to simultaneously solve the "selection of variables" and the "parameter estimation" problems in regression. The basic LASSO regression model imposes a fixed penalty rate on each slope coefficient (so that the penalty is proportionate to the magnitude of the coefficient), whereas the adaptive LASSO regression model involves adding adaptive weights to the penalties for the different slope coefficients.
The asymptotic properties of the adaptive LASSO model are discussed in Zou (2006). This shows how the weights in the adaptive model can be set in order to give some desirable asymptotic properties that are absent from the basic LASSO model. As the number of data points gets larger and larger, the adaptive weights for the zero coefficients explode to infinity (and thereby impose a boundless penalty on these coefficients), while the adaptive weights for the non-zero coefficients converge to a finite upper bound (and thereby impose only a finite penalty that is outweighed by the log-likelihood part of the optimisation). Zou shows that under the adaptive method shown in that paper, the identified set of non-zero coefficients converges to the true set of non-zero coefficients (i.e., the selecetion of variables is consistent) and the estimator for the non-zero coefficients has an asymptotic normal form. The former property ensures that the selection of variables is accurate over large samples, and the latter property ensures that one can obtain reasonable large-sample approximations for the distribution of the coefficient estimators.
I see no particular reason that the adaptive LASSO model would not be useful in cases where you have a number of binary variables. When penalising binary variables we sometimes scale these explanatory variables to have equal sample variance prior to fitting. In any case, while there may be other methods that outperform adaptive LASSO in particular cases, it has some useful consistency properties that ensure good large sample performance. I will leave it to others to propose any alternative models that they believe would have better performance.
-
$\begingroup$ it should help to call that property with its name: "oracle property". a number of methods have it, including adaptive LASSO and SCAD, it is, however, limited by the assumption of linearity of the true model $\endgroup$– carloSep 9, 2020 at 7:16
$\begingroup$
$\endgroup$
6
Not necessarily. Search around on feature selection and model selection. Model selection is not a solved problem and it is unlikely to be solved since it is NP-hard.
In my own experience, I have seen the LASSO sometimes select poor or even insanely wrong models. That is not restricted to the LASSO. Ridge regression, stepwise selection methods, searches using AIC and BIC, random forest, SVMs, ... I have seen them all fail spectacularly.
I know you want a slick answer that sounds like it will work; however, this is one of the areas of statistics where we really have to work hard and use our experience. Furthermore, you are really exposing yourself to Simpson's Paradox and structural breaks if you just grind the data through a method instead of looking at it carefully with simpler approaches first.
One of my favorite assignments for students is to give them some data on petroleum products. If you use the LASSO, ridge regression, SVMs, or assume a cointegrating relationship, the data give you a model that is absurd -- as in completely unrelated to the reality of refining processes. Furthermore, those models perform horribly out-of-sample; you would be better off without a model. With some theory to guide the modeling and looking at the data in smaller time groupings, however, the expected structure emerges.
-
$\begingroup$ I'm using multiple approaches: LASSO, multiple regression, stepwise regression (using AIC and BIC) and best subsets regression. Does that seem exhaustive? $\endgroup$– DaveAug 28, 2020 at 21:53
-
3$\begingroup$ Could throw ridge in there too. That is exhaustive and yet... you still may not find a good model. An example: I once modeled returns for a US real estate firm. Ridge, LASSO, and everything else suggested a US real estate index and 10-year US interest rates... and the Canadian stock market, US natural gas, and the USD/CAD exchange rate. Stepwise with AIC and BIC gave even bigger models. Cross validation didn't fix a thing. The only solution was using some theory and domain knowledge -- which yielded the best model by far. So I would think about your data rather than just trying many methods. $\endgroup$– kurtosisAug 29, 2020 at 0:35
-
-
$\begingroup$ @BenBolker True, although random forest and other algorithms that use classification in some way suffer in financial contexts. Since there are a number of different classifiers but they are somewhat isomorphic, any prediction which could be found via classification is typically competed away. This induces a skewness in returns and leaves most predictions needing to consider the magnitude of the effect as well as the direction. $\endgroup$– kurtosisSep 10, 2020 at 21:41
-
$\begingroup$ @BenBolker Would random forests give me information on the relationships between predictors and the DV? $\endgroup$– DaveSep 15, 2020 at 18:29
$\begingroup$
$\endgroup$
The answer depends on whether you are restricting yourself to the class of linear models, which I will define as something with the form: \begin{align} y_i &\sim \mu_i \\ g(\mu_i) &= X_i\beta. \end{align} Further, let's denote the sample size by $n$ and the number of predictors/variables by $p$.
Case 1: Linear model
If you have a large sample, then simple, un-regularized regression will converge to the true values of $\beta$ if $p$ remains small (say 40). This naturally begs the question: what counts as a large sample? Well, it depends. If there's no severe collinearity and all the variables have decent representation (for example, we don't have binary variables with only one 1 and all other 0), then a few thousands would be considered large.
However, when you do have samples of this size, then typically statisticians would consider modeling possible non-linearity in the data. For example, one could include interaction terms or polynomial terms, which could increase your number of variables massively if a large number of these are considered. One could then use LASSO or better still, Elastic Net, to regularize the model, since LASSO is simply a special case of Elastic Net. Note that neither the LASSO nor the Elastic Net (EN) has the oracle property, which means there's no guarantee that the estimated $\beta$ converges to their true values with infinite sample size (although adaptive LASSO does). If interpretation is important, as opposed to prediction, then this may put some off using these techniques. Moreover, it may be possible that some interaction effects are retained while the main effects are excluded, which can further hamper interpretation, although one can impose constraints to prevent that.
However, in case where the sample size is not large or when you want to consider a large number of possible non-linearities (i.e. you have large $p$), then the lack of the oracle property is arguably irrelevant, and I would argue that the EN is a reasonable choice. By "reasonable" I mean a reasonable choice over alternatives such as best-subset/stepwise regression, which are simply coarser forms of regularization. On the other hand, there are an infinite number of ways one could regularize a linear model. There is simply no one method which is the "best" in all cases.
Case 2: Non-linear model
Because of possible non-linearities, one could consider non-linear approaches such as SVM/SVR or random forest. One can use approaches such as permutation or dropping the variables to investigate the importance of the variables concerned. See here for some intuition.
Overall
Note that whether in the linear or the non-linear model case, whether a variable is important in the prediction of the outcome depends critically on the target population. These methods all suppose that the target population is the same as the source population, i.e. the population from which you derived the sample. A variable that is unimportant in the sample can turn out to be hugely important in the target. This kind of information will require domain knowledge. It also implies that ranking variable importance in terms of some derived statistics will always have some serious limitations.
