This is regarding a binary classification with logistic regression with one independent variable (IV) and one dependent variable (DV). The IV seems to be distributed normally, the residuals have a random distribution, and AS FAR AS I KNOW, based on my domain knowledge and looking at a plot, it should affect the DV in a linear manner, and it is definitely a monotonic relationship. So basically, it should be a very straight forward relationship. There are 1100 datapoints.

When I run the regression and check the predictions with 10fold cross validation, it predicts at about 51% accuracy or even just randomly. There is a lot of variance so the best we can hope for is maybe 54%. It is worth noting the model predicts a success at about 4 to 1 rate when the true rate is close to 1 to 1.

Question 1: At this point, if you were trying to improve this regression, what would you try to do? What do you think may be the problem?

So I tried various things like adding different IV's, trying random transformations, and had no success. Then I started clustering the data and only selected ABOUT half the datapoints which most closely match the test case I am trying to predict. This improved the accuracy greatly.

Question 2: Why would clustering help in such a simple, basic, regression situation? Is there a more philosophically sound, more effective way than clustering to set up this regression?

  • $\begingroup$ I'm not quite sure you mean when you say "The best we can hope for is maybe 54%". Assuming you're referring to correctly predicting the class labels and that the outcomes aren't nearly 50/50 coin flips, you can probably do better than that without a model at all. For example, if the proportion of $1$s in the data set is $p>1/2$ (if $p<1/2$ just flip the labels). Then you will get a prediction accuracy of $p$ just by guessing everything is a $1$. Maybe I've missed your point.. $\endgroup$ – Macro Jul 25 '13 at 17:15
  • $\begingroup$ Marco, in this situation p is designed to equal 50%. I try to find "hidden information" to improve accuracy prediction above 50% $\endgroup$ – appleLover Jul 25 '13 at 22:17

What you call "clustering" is also known as local regression, kernel regression or local likelihood smoothing. The overall framework is generalized additive modelling, and the definitive textbooks are Hastie & Tibshirani (1990) Generalized Additive Models, and Wood (2006) Generalized Additive Models: An Introduction With R.

GAMs extend on GLMs (including logistic regression) by allowing nonlinear trends to enter the model in a data-driven way. You can include such nonlinear trends manually via transformations, eg polynomial terms or spline terms, but this usually requires examining the data beforehand. This can be tedious if you have many variables and/or they're correlated with each other. The fact that using a local fit improved your model suggests that the relationship between your IV and DV is nonlinear.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.