Is this overfitting?

The Use Case: We are given three unique, 'ground truth' binary training patterns (not patterns with 'noise'). A machine is to be trained with these three vectors. The requirement is that once trained, the machine be used in a game, such that: players in advance don't know exactly what each of the 'true patterns' are, but will try to independently 'derive' at least one of them, and then give their attempt to the machine to judge. When the model receives the player's attempt, it will return to the player the closest matching 'true' pattern to what the player has input. The player may have input a partial section of a 'true' pattern, or a section with some additional bits (ie 'noise'), but either way the model will effectively say 'Ah, I see what you've done there, here is what your pattern should be'.

The Question: I have created a Restricted Boltzmann Machine with 9 binary inputs and 3 hidden units that achieves the objective of the use case. The machine converged to 0 error on the training vectors, and accurately provides the 'nearest match' to the player's inputs, which the machine has not seen before. So which of these statements is correct:

a) The machine IS overfitting, but nevertheless is providing a useful predictive/classification capability in the context of the use case

b) The machine is NOT overfitting, because if it was, it would not be accurately returning the nearest trained 'truth' matches to partial/incomplete/noisy input patterns it has not previously seen

If you continue to achieve zero error on new input data with the same original model/algorithm it means the phenomenon you're modeling is deterministic.

• So is this 'deterministic learning?' Overfitting implies the model has not 'learned' and so does not produce meaningful results. But here we have 'learning' because even though the model outputs same 'baseline shapes' it has been trained on, it outputs the 'closest matching' one to any partial/incomplete/noisy shape a user inputs. So, it is doing something meaningful. Would it be correct to say then that even though we have zero training error and zero validation error here (the model never chooses an incorrect baseline shape per user input), 'overfitting' does not apply in this case? – samiant Jul 31 '20 at 9:48
• Most real-world situations have random noise and are never perfectly predicted. Overfitting happens when training error is so low that validation error starts to increase, so technically speaking, I don't think you've "overfit," because your validation error has yet to increase. – Alex Jul 31 '20 at 13:28

If you get zero error in any machine learning problem, it means you are overfitting. Getting a zero error means your model has figured out the true function which cannot happen as there will always be indeterministic noise present even if you have zero bias and variance which is a highly improbable scenario. This is all part of machine learning theory and how generalization happens. In short, your model hypothesis will always approximate the true function which is always unknown in machine learning and hence cannot achieve 100% results.

• Why assume that it is never possible to figure out a true function? What of the machine learning problem where you need to predict values in the following sequence: $3,3,3,3,3,3,3, \cdots$. Is it really overfitting if the machine achieves zero error in this case? – Ben Jul 30 '20 at 4:57
• Apparently machine learning is stochastic, not deterministic and stochastic is having a random probability distribution or pattern that may be analysed statistically but may not be predicted precisely. – Vivek Jul 30 '20 at 5:52
• Reality dictates what is stochastic or deterministic, not the disciplines of machine learning or statistics. The latter are merely tools for inference, and ideally they should work in both cases (but certainly they should at least be able to predict a simple deterministic series well). – Ben Jul 30 '20 at 6:59
• I strongly disagree, if a process is stochastic then machine learning is used to model it. However, if a process is deterministic then using machine learning in this case will be equivalent to data memorisation and there will be no actual learning involved. Reference: youtube.com/watch?v=MEG35RDD7RA&t=3693s – Vivek Jul 30 '20 at 7:21
• You appear to be conceding that machine learning can be applied to a deterministic process. Would you agree that it can predict this kind of process correctly with sufficient data? If so then presumably it is possible to get zero error without overfitting. – Ben Jul 30 '20 at 7:28

Conceptually, the accuracy of your model is a function of these factors:

1. complexity of the model
2. number of samples in the training set
3. variety of training samples (e.g. noisy samples VS repetitive samples)
4. number of samples in the validation set
5. variety of validation samples (e.g. noisy samples VS repetitive samples)

Right now you have 100% classification accuracy both on the training set and validation set. But you can study how accuracy degrades, if you artificially make any of these factors worse (e.g. reduce the training set size).

You can also draw accuracy curves by varying one factor at the time, while keeping the other 4 fixed. This way you will gain a deep understanding of your problem and how your model solves it.

The success of your model in real life will depend on how those 5 factors score in your real life scenario. For example, you might have to live with a small and messy training set. Or you might be lucky and sit in a very predictable scenario with plenty of historical data.

In all cases, understanding the sensitivity of your model in the depth, gives you the confidence that you know what your are doing 🙂 when money is at stake.