I have a ($k$-class) classification problem, with of the order of 100 real-valued predictors, one of which appears to have much more explanatory power than any of the others. I'd like to get deeper into the effects of the other variables. However, standard machine learning techniques (random forests, SVMs, etc) seem to get swamped by the one strong predictor and don't give me much interesting information about the others.
If this were a regression problem, I would simply regress against the strong predictor and then use the residuals as inputs for other algorithms. I don't really see how this approach can be translated to a classification context though.
My instinct is that this problem must be reasonably common: is there a standard technique for dealing with it?