1
$\begingroup$

I want to split my main dataset in two part, training dataset and test dataset.

In the past i read somewhere (which unfortunately i could not find exactly where was that), that when splitting my main dataset in train and test i should pay attention that two observation of same origin (correlated data) should not placed in train and test group.

For my question i want to ask, why i should not place 2 observation of same origin in both test and train group (imagine i used k-fold for splitting my main dataset)? what problem it would cause? it cause error in classification or some overfitting/underfitting problem?

$\endgroup$
2
$\begingroup$

Suppose you have a dataset with credit card transactions, with binary labels indicating whether they were fraudulent or genuine. I suppose the user ID or card ID of a transaction could be viewed as its ''origin'' in the context of your question.

Now, suppose you have a single user / card ID which was used in multiple transactions. If it was a fraudster, it's likely they have multiple fraudulent transactions in the dataset (assuming they were not immediately caught and blocked). If you put some of these transactions in the training data, and some other transactions of the same user in the test data, those test cases may be too easy to detect for a trained model.

As an extreme case, suppose we include the card ID itself as a feature when training our model. It can simply memorize that that card ID was associated with fraudulent transactions in the training data, and have an unrealistically easy time detecting and flagging them in the test data. This model would start performing significantly less well in the future, when all those card IDs it has memorized have been blocked.

Of course, this is an extreme example, you probably wouldn't want to include the raw card ID as a feature. The issue can still appear in a less extreme manner in realistic cases though (for example, users may be memorized and recognized based on pattersn in their behaviour which are encoded in features).

$\endgroup$
  • $\begingroup$ Thanks a lot @Dennis Soemers, so in order to summary your answer, you say in a case of correlation between test and train dataset, classifier wrongly produce a good performance . $\endgroup$ – maia Jan 3 '18 at 12:53
  • 1
    $\begingroup$ @maia Yes, or more precisely, it can appear to be better than it is. Suppose you train two different models A and B. Somehow, A manages to ''cheat'' by memorizing card IDs like I described in the answer, and B doesn't cheat and only uses ''proper features''. If you then evaluate their performance on a test dataset, A may have a better performance than B on that test dataset, but B would probably perform better than A when actually used ''in the real world'' afterwards $\endgroup$ – Dennis Soemers Jan 3 '18 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.