12
$\begingroup$

I would like to know the differences between Randomized Logistic Regression (RLR) and plain Logistic Regression (LR), therefore, I am reading a paper "Stability Selection" by Meinshausen, et al.; however I do not understand what RLR is and what the differences between RLR and LR are.

Could someone point out what I should read to understand RLR? Or is there a simple example to start with?

$\endgroup$
  • 1
    $\begingroup$ RLR is not a standard term. Please define the method. $\endgroup$ – Frank Harrell Jan 2 '15 at 14:44
  • $\begingroup$ Thank You @FrankHarrell ... The method is coming from a scikit learn library. $\endgroup$ – Hendra Bunyamin Jan 2 '15 at 15:33
  • $\begingroup$ Now that there is a new stack exchange site for machine learning/Big Data, perhaps this question belongs over there. $\endgroup$ – Placidia Jan 2 '15 at 15:49
  • 4
    $\begingroup$ @Placidia That's a good suggestion. However, your very own answer shows why this question belongs here: we are better able to provide a balanced perspective that accurately characterizes and compares both the statistical and ML aspects of the question. Although it is possible that someone on the "data science" site could contribute such an answer, my experience there is that it would be unlikely. $\endgroup$ – whuber Jan 2 '15 at 15:57
  • 3
    $\begingroup$ I am stunned that the new site is call data science, which is more than half about statistics, which is what this site is about. $\endgroup$ – Frank Harrell Jan 2 '15 at 17:43
17
$\begingroup$

You might want to check out this reference . Sci-kit learn implements randomized logistic regression and the method is described there.

But to answer your question, the two methods differ largely in their goals. Logistic regression is about fitting a model and RLR is about finding the variables that go into the model.

Vanilla logistic regression is a generalized linear model. For a binary response, we posit that the log odds of the response probability is a linear function of a number of predictors. Coefficients of the predictors are estimated using maximum likelihood and inference about the parameters is then based on large sample properties of the model. For best results, we typically assume that the model is fairly simple and well understood. We know what independent variables impact the response. We want to estimate the parameters of the model.

Of course, in practice, we don't always know what variables should be included in the model. This is especially true in machine learning situations where the number of potential explanatory variables is huge and their values are sparse.

Over the years, many people have tried to use the techniques of statistical model fitting for the purpose of variable (read "feature") selection. In increasing level of reliability:

  1. Fit a big model and drop variables with non-significant Wald statistics. Doesn't always produce the best model.
  2. Look at all possible models and pick the "best". Computationally intensive and not robust.
  3. Fit the big model with an L1 penalty term (lasso style). Useless variables get dropped in the fit. Better, but unstable with sparse matrices.
  4. Randomize method 3. Take random subsets, fit a penalized model to each and collate the results. Variables that come up frequently are selected. When the response is binary, this is randomized logistic regression. A similar technique can be pulled with continuous data and the general linear model.
$\endgroup$
  • 1
    $\begingroup$ +1 It is a pleasure to see such a well-articulated, readable, informative survey of a general methodology. $\endgroup$ – whuber Jan 2 '15 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.