Multiple classifiers always wrong one the same examples: could it be exploited?

Question

I trained a different number of classifiers on a problem (logistic regression, KNN, neural networks, svm...). The training happened with double cross validation and all the gold standards, so I'm pretty confident they are good classifiers, in the sense that they do not overfit. If I examine the confusion matrix each one of them generates on a test set, I see they all make almost the same mistakes (numbers in the confusion matrix are roughly the same for each one of them).

Can this common pattern in how they classify be exploited in some way? For example, I was thinking of stacking one of such models with other models that have decorrelated predictions, something along these lines. Does this make any sense? Do you have better ideas to propose?

Answer 1

See https://en.wikipedia.org/wiki/Boosting_%28machine_learning%29

You are close to rediscovering boosting, an ensemble learning idea consisting in learning multiple weak/simple algorithms, but reweighting the data, so the examples badly classified for the first algorithms, are given higher weigh, so later algorithms maybe can find patterns in the data that help with difficult examples. Search this site for boosting.