1
$\begingroup$

I have been thinking about this for a few so I would like to hear some opinions. It could be complicated to explain so I will update the question if there is something that its not clear.

Imagine I have a tabular dataset looking like this:

Code, F1, F2, F3, F4, TARGET
 01,  15, 23, 43, 12,   10

Suppose I plot the distribution for CODE=01 and all the targets are between 10 and 12. This means that, if I dont take any other feature into account, If I have a neural network It could learn that when CODE == 1 prediction is 11 and it has maximum 10% error.

My train dataset has some samples as the one mentioned in the code snip, but my test ALSO has similar samples.

Training_set
Code, F1, F2, F3, F4, TARGET
 01,  15, 23, 43, 12,   10
 01,  17, 32, 23, 8,    12

Test_set
Code, F1, F2, F3, F4, TARGET
 01,  21, 12, 43, 12,   10
 01,  23, 2,  12, 43,   11

So anyways the weight of my NN says that when input CODE=1, prediction is going to be 11. Error is ''good'' in both train and test and even validation. Kfold says everything is good. But the truth is that my NN has just 'learned' a simple rule, CODE=1, PREDICTION=11. It works in the real world because Im always receiving some CODES=1, with should be predicted between 10 and 12. Would you considered this overfitting?

Now imagine that NN is using all the features.

Training_set
Code, F1, F2, F3, F4, TARGET
 01,  15, 23, 43, 12,   10

Test_set
Code, F1, F2, F3, F4, TARGET
 01,  15, 23, 23, 8,    10

As you can see there is a huge common part between samples. Would you consider this as a duplicate? If I do some contribution analysis and I see that my NN is using basically using the firsts 3 features (code, f1, f2) to take a decision and this information WAS in my training set.. is it still valid? it is overfitting? should know I consider it as a duplicate?

Just another example: Imagine I'm trying to identify FACES, I want to know if a photo has a face. Then in my test_set I put a photo of a friend which is already in the training BUT it is a different photo. In this case I could try simply with other faces and see the results, but imagine that in the real world Im gonna use ONLY photos on my friend so overfitting will be ok as long as I detect that?

Thanks in advance.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The two matters:

  1. Overfitting and
  2. using only some of the given features

are entirely different. For e.g. finding that you need only a subset of the features to do prediction is totally fine, happens a lot, and doesn't mean that you are overfitting. Overfitting is when you have bad generalization because you trained an overly complex model on too small a dataset.

I am not sure I interpret your last example properly, but for good generalization, your training set needs to be representative of your target population, your test set. Of course, it doesn't have to learn stuff that it doesn't need later when applied "in the real world".

Improve this answer
$\endgroup$
4
  • $\begingroup$ Yes. I mean, I completely understood the difference between one and the other. My question is more related to: what is a duplicated? coul a 'partial' duplicated be a risk in case one is the training and the other in the test? (my second example) $\endgroup$
    – Sergiodiaz53
    Commented Mar 17, 2022 at 15:05
  • $\begingroup$ I am not sure I understand: Two measurements are duplicates when both have exactly the same features. Duplicates and "partial duplicates" are fine and happen all the time, in training and/or test set. They are not harmful. $\endgroup$
    – frank
    Commented Mar 17, 2022 at 15:11
  • $\begingroup$ Well but if you have duplicates on your training you are, somehow, giving them more importance during the training (the algorithim is seeing those examples MORE than the not duplicates one). Now if you have overfitted ur network and you have 1 duplicate sample in your test that it is ALSO in your train (because it appears two times in the dataset) then you are going to predict perfectly that test sample because you have learnt the prediction before. Dummy case will be using the same train as test imagining all your samples are duplicated and you are very unlucky. $\endgroup$
    – Sergiodiaz53
    Commented Mar 17, 2022 at 16:11
  • 2
    $\begingroup$ Sometimes the probability distribution is such that you end up with some equal cases in the training and in the test set, and that would still be a genuine, independent representation of the population in both datasets. If you have sampled your training dataset independent from your test dataset, there is no leaking of information. And that is all that matters. $\endgroup$
    – frank
    Commented Mar 17, 2022 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.