In the Figure1 below, the Training data is "red" and and Test data is "black". Is it correct to assume that any binary classifier will not generlize well? (keep in mind that the colors in the figure are not class labels..)

My rationale is that the both datasets have a different underlying mathematical structure (as visualized by tSNE), and thus a classifier trained in either one of these datasets will not perform well if tested on the other.

Train and Test with tSNE

EDIT1: As a followup analysis (cannot post image due to low reputation), I individually filtered out all variables whose distribution is different in both datasets. The resulting figure appears much more "well mixed" (i.e. a good mix of red and black points). This "reduced" dataset is thus more suitable for classification, correct? Nonetheless I wonder if all classifiers can implicitly handle such covariate shifts?

  • 1
    $\begingroup$ Is it correct to assume that any binary classifier will not generalize well? -- It's not necessarily true. The difference between your classes could be orthogonal to the train/test (red/black) difference. $\endgroup$ – amoeba Jul 3 '17 at 10:36
  • $\begingroup$ @amoeba, I think I see what you mean. Though, ideally, I expect a practically useful classifier to be trained (and tested) on data with a shared "true" underlying distribution. If this assumption is violated (often in the real world), then we need to account for covariate shift. Correct? PS: I've updated my original post $\endgroup$ – nafs_nafs Jul 3 '17 at 11:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.