The Differences between Randomized Logistic Regression and Plain-Vanilla Logistic Regression

Question

I would like to know the differences between Randomized Logistic Regression (RLR) and plain Logistic Regression (LR), therefore, I am reading a paper "Stability Selection" by Meinshausen, et al.; however I do not understand what RLR is and what the differences between RLR and LR are.

Could someone point out what I should read to understand RLR? Or is there a simple example to start with?

Answer 1

You might want to check out this reference . Sci-kit learn implements randomized logistic regression and the method is described there.

But to answer your question, the two methods differ largely in their goals. Logistic regression is about fitting a model and RLR is about finding the variables that go into the model.

Vanilla logistic regression is a generalized linear model. For a binary response, we posit that the log odds of the response probability is a linear function of a number of predictors. Coefficients of the predictors are estimated using maximum likelihood and inference about the parameters is then based on large sample properties of the model. For best results, we typically assume that the model is fairly simple and well understood. We know what independent variables impact the response. We want to estimate the parameters of the model.

Of course, in practice, we don't always know what variables should be included in the model. This is especially true in machine learning situations where the number of potential explanatory variables is huge and their values are sparse.

Over the years, many people have tried to use the techniques of statistical model fitting for the purpose of variable (read "feature") selection. In increasing level of reliability:

1. Fit a big model and drop variables with non-significant Wald statistics. Doesn't always produce the best model.
2. Look at all possible models and pick the "best". Computationally intensive and not robust.
3. Fit the big model with an L1 penalty term (lasso style). Useless variables get dropped in the fit. Better, but unstable with sparse matrices.
4. Randomize method 3. Take random subsets, fit a penalized model to each and collate the results. Variables that come up frequently are selected. When the response is binary, this is randomized logistic regression. A similar technique can be pulled with continuous data and the general linear model.