I am using a random forest for a 2 class classification problem. But eventually using probability of class "1" returned by the model for my task and not the label. I get AUC of about 70%
Then I compare the probability with the real world value and measure the difference (residual). Then I build a regression random forest model to predict the residual given the same features! This seems to be a weird idea but I tried it. Then I correct the probabilities returned by the first model with the output of the 2nd model and this improved performance in the "test set" Significantly! The 2nd model explains 85% of the variability.
What does this mean? Why is the first model not accurate enough? Even those the same features are used in the 2nd model, it improves performance.
Somehow, the model the predict the residual of the classification model has a higher performance compared to the classification model itself. And both models use the same features.