Usually, the validation error is higher than training error, but are there any cases when they are equal?
- Reason 1: the model is underfitted, i.e. it has a high bias:
Reason 2: the model is near perfect.
Reason 3: the training set is very similar to the validation set, e.g. if some data from the validation set have leaked into the training set:
- Reason 4: if using a neural network, the training have been prematurely stopped:
Ideal, anything can happen depends on the distribution of training data and testing data. I can easily craft an example that make training error and testing error to any value you want.
In real world, if you build model very carefully and you have large amount of data. The testing error should be very close to training error. This is because we are trying to use math to describe a physical relationship. If the model is good enough, it should be applicable in future.