I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of the training set and testing set are similar.
And I search it and find a blog which check the similarity of distributions by converting it into a binary classification problem. If it gets a high AUC, the distribution of the training set and testing set must be different. And the idea he gives as follows:
If there exists a covariate shift, then upon mixing train and test we’ll still be able to classify the origin of each data point (whether it is from test or train) with good accuracy.
But I still can't understand why he can check the similarity of these two distributions in this way.
And are there other ways to check the similarity of it?
It will be appreciated, if anyone could help me.