1
$\begingroup$

I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of the training set and testing set are similar.
And I search it and find a blog which check the similarity of distributions by converting it into a binary classification problem. If it gets a high AUC, the distribution of the training set and testing set must be different. And the idea he gives as follows:

If there exists a covariate shift, then upon mixing train and test we’ll still be able to classify the origin of each data point (whether it is from test or train) with good accuracy.

But I still can't understand why he can check the similarity of these two distributions in this way.
And are there other ways to check the similarity of it?
It will be appreciated, if anyone could help me.

$\endgroup$
0
$\begingroup$

If there is no covariate shift and the distribution of training and test data is the same, it will be very hard to correctly classify which data points come from training and which ones are from test. In this case the classifier would not be much better than random guessing because it doesn't have any information available to make a better guess.

If the classifier manages a much better prediction rate than random guessing there must be some information where test and training data differ. So there must be some kind of difference between the underlying distributions that the classifier picked up.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.