I trained a different number of classifiers on a problem (logistic regression, KNN, neural networks, svm...). The training happened with double cross validation and all the gold standards, so I'm pretty confident they are good classifiers, in the sense that they do not overfit. If I examine the confusion matrix each one of them generates on a test set, I see they all make almost the same mistakes (numbers in the confusion matrix are roughly the same for each one of them).
Can this common pattern in how they classify be exploited in some way? For example, I was thinking of stacking one of such models with other models that have decorrelated predictions, something along these lines. Does this make any sense? Do you have better ideas to propose?