This is regarding a binary classification with logistic regression with one independent variable (IV) and one dependent variable (DV). The IV seems to be distributed normally, the residuals have a random distribution, and AS FAR AS I KNOW, based on my domain knowledge and looking at a plot, it should affect the DV in a linear manner, and it is definitely a monotonic relationship. So basically, it should be a very straight forward relationship. There are 1100 datapoints.
When I run the regression and check the predictions with 10fold cross validation, it predicts at about 51% accuracy or even just randomly. There is a lot of variance so the best we can hope for is maybe 54%. It is worth noting the model predicts a success at about 4 to 1 rate when the true rate is close to 1 to 1.
Question 1: At this point, if you were trying to improve this regression, what would you try to do? What do you think may be the problem?
So I tried various things like adding different IV's, trying random transformations, and had no success. Then I started clustering the data and only selected ABOUT half the datapoints which most closely match the test case I am trying to predict. This improved the accuracy greatly.
Question 2: Why would clustering help in such a simple, basic, regression situation? Is there a more philosophically sound, more effective way than clustering to set up this regression?