I have some experience working with classification, and in those instances we always use a training and a test set (and possibly validation sets).

However, I'm currently facing a different problem. I want to test how well I can combine different predictors, and find the relative importance of each of them. I have a binary dependent variable and am therefore currently using logistic regression for this. My question is, is it weird to run logit on all the data, and then talk about the goodness of fit and coefficients? I've read some literature on the subject, and I think I'm simply having a hard time wrapping my head around the fact that logit isn't (just) a classifier, but I just want to make sure that it makes sense to use it to investigate how my predictors behave on all of my data.


2 Answers 2


A quick answer to your question, "Is it weird...?" is that it would be hugely inefficient and likely erroneous to do what you're proposing. I want to warn you against using all of your data and variables in a single pass as not being terribly informative and, depending on how much data and how many variables you have, likely computationally infeasible.

Your question is general enough, though, that there can't be a single, "right" answer. Basically, it sounds like you're asking about variable (coefficient) selection, relative importance, stability and performance out-of-sample. A host of issues that are quite distinct and have been boiled down to out-of-sample predictive accuracy for all too many analysts.

Let's divagate: Leo Breiman developed his random forests method as an approximating workaround and solution to many of these concerns although he way over-emphasized the importance of predictive accuracy (alternatively, minimizing the MSE). RFs were originally intended to answer the criticisms of his CART technique that one-off, stand-alone trees were quite unstable out-of-sample. Breiman wrote later papers in which he generalized the resulting solutions to variable selection. His method involved building many smaller, bootstrap sampled trees (from the original, complete raw data) which represented randomly drawn subsets of data and candidate predictors. In the process, one lost the "tree" structure inherent to a single solution, but gained accuracy via the ensemble approach to aggregating the predictions.

The simple fact is, and this hasn't been considered by many statisticians, the RF structure is applicable to any multivariate tool, not just classification trees. Obviously, this would include logistic regression.

Breiman did his work in the 90s on a single CPU when "massive" meant a few thousand candidate predictors and a few gig of data, the physical limitations of a machine in terms of I/O and RAM were frequently quickly reached. Today, a variety of so-called "divide and conquer" routines have been developed for massively parallel, multi-core machines in the presence of tens of thousands or even hundreds of thousands (millions?) of candidate predictors and terabytes of data. These new routines can run millions of "mini-models" in a matter of a few hours, depending on the installation. For the most part, they are extensions of Breiman's basic RF framework. Currently, there are no good, inexpensive IT solutions to analyzing petabytes of information.

These D&C algorithms are technical solutions only. Other evaluatory considerations include those already mentioned as well as others not explicitly listed. See this link for some wider considerations:

Should predictive accuracy or, alternatively, minimizing the MSE, be reconsidered?

In addition, here's a recent review of D&C routines:

A Survey of Statistical Methods and Computing for Big Data by Wang, Chen, Schifano and Win (2015).

  • $\begingroup$ A quick comment (haven't looked at your sources yet, on mobile): Thanks for the detailed response. It is indeed kind of variable selection, and possibly performance in-sample. Thanks for a good read overall! $\endgroup$
    – keyser
    Jun 13, 2015 at 18:15
  • $\begingroup$ And a side note: I have about 6 predictors and 500k samples $\endgroup$
    – keyser
    Jun 13, 2015 at 18:18
  • $\begingroup$ @keyser In that case, forget everything I said and just use the Lasso technique for variable selection. $\endgroup$ Jun 13, 2015 at 18:24

It's not weird, I think. I know in r, for example, there's a package that gives you the relative importance of all of the predictor variables.

Furthermore, finding the "goodness" of coefficients is very common, as this is one way to find which coefficients are insignificant or obsolete.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.