I trained a different number of classifiers on a problem (logistic regression, KNN, neural networks, svm...). The training happened with double cross validation and all the gold standards, so I'm pretty confident they are good classifiers, in the sense that they do not overfit. If I examine the confusion matrix each one of them generates on a test set, I see they all make almost the same mistakes (numbers in the confusion matrix are roughly the same for each one of them).

Can this common pattern in how they classify be exploited in some way? For example, I was thinking of stacking one of such models with other models that have decorrelated predictions, something along these lines. Does this make any sense? Do you have better ideas to propose?


See https://en.wikipedia.org/wiki/Boosting_%28machine_learning%29

You are close to rediscovering boosting, an ensemble learning idea consisting in learning multiple weak/simple algorithms, but reweighting the data, so the examples badly classified for the first algorithms, are given higher weigh, so later algorithms maybe can find patterns in the data that help with difficult examples. Search this site for boosting.

  • $\begingroup$ Yeah, I'm aware of gb, but actually it performs like the others. I'm thinking that I sort of hit the "upper bound" of performance, given that the outcome that I'm predicting is also determined by features that are not included on purpose (it's a medical problem, so I was told not to use certain attributes). $\endgroup$ – mp85 Nov 3 '16 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.