I have a number of predictors to use for a binary (Classes 0 and 1) classification task. Let us call them $x_1, x_2, x_3, ... x_n$. The way these are calculated, my naive heuristic assumption is that $S = \sum_{k=1}^n x_k$ should predict Class 1. That is higher $S$ means it is more likely that it is Class 1.

Since this did not perform as well as I had hoped, I want to show that my scores $x_k$ can collectively achieve a better performance, than, say some other baseline methods and even my own single scores such as $x_1$. Due to the way I justify my calculations of $x_k$, I do not want to use a black-box technicque such as SVM or Random Forest. More concretely, if I use a very complex model, there is a greater burden on me to prove that I am not over-fitting or just getting as lucky as any other combination of various predictors used in the field.

I first tried to use logistic regression and its performance was not very spectacular. I then proceeded to use MARS models, and I got much better classification performance (using Precision-Recall and ROC curves). But scientifically, I am not sure I can simply justify using MARS models and comparing it to logistic regression. So I want to know which models are the logical extensions to use after logistic regression to add a little more complexity. I have thought of splines, MARS, additive models, adding binary interactions $x_i x_j$ as inputs to the logistic regression. I have planned to use PR and ROC curves on cross-validation data sets to do the comparisons. For logistic regression, one can use model likelihood as a measure but are there any other measures to use across models such as these?


The right thing to do after a logistic regression is to check whether your classification performance suffers from high bias or from high variance. That will tell you if you are underfitting or overfitting, respectively. If you have a high variance problem, you'll want to regularize your regression. If you have a high bias problem, you'll want to add more features, such as binary interactions. Remember to evaluate on a separate validation set, not on training data.

Precision and recall are great measures to use even for logistic regression. You may want to combine them into an F-score so that you have a single metric to optimize.

See Andrew Ng's lectures at ml-class.com for further information on logistic regression, regularization, bias, and variance.

  • $\begingroup$ Thanks! That got me thinking, that if possible perhaps I should use a Bayesian framework so I don't have to run these bias-variance plots repeatedly. I will be applying the same algorithm to many data sets and I think because of the amount of data and noise, the optimal bias-variance trade off might be different. Once I have determined it to be high bias, what models should be tested next?. Are interaction terms the only option? Is there bayesian splines, MARS, etc.? Any thoughts from you or anyone else? $\endgroup$ – highBandWidth Dec 31 '11 at 14:44
  • $\begingroup$ Interaction terms are not your only option; any transformation of a term or combination of terms is possible, e.g., log(x1)^2*exp(x2). The question about Bayesian approaches to the lots-of-datasets problem is really interesting; try to post that one separately. $\endgroup$ – Jack Tanner Jan 2 '12 at 3:22
  • $\begingroup$ Can you say any more about how exactly you'd determine whether bias or variance is the problem? With a binary outcome the mean and variance are linked so I'm curious how you'd quantify this. $\endgroup$ – Macro Jan 3 '12 at 3:53
  • $\begingroup$ Macro: I second Jack's recommendation to watch Ng's lectures. Having just finished his online class, he goes in to great detail on bias vs variance. $\endgroup$ – JonnyBoats Jan 3 '12 at 4:20
  • $\begingroup$ To tell whether a problem is high bias or high variance, per Ng, downsample your training set, train your regression, and evaluate error on the training set and on a validation set. Then keep increasing your sample size, re-training and re-evaluating until you're "sampling" 100% of the training set. If error on the validation set asymptotically tends just a little higher than on the training, you have high bias; add more features. If error on the validation set tends much higher than on the training, you have a high variance (overfitting); use regularization and try to get more data. $\endgroup$ – Jack Tanner Jan 3 '12 at 6:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.