Training vs test accuracy trade-off

Question

Say I am training a neural network for image classification of cats and dogs, in a dataset of 1000 for example. I train the network on 800 examples, and then I test it with the remaining 200 examples, and I get training and test accuracies of 94% and 73% --> 21% gap

Applying regularization should fix this train-test accuracy gap up to some extent. Would it be right to assume that there is always the possibility of a trade-off between training accuracy and test accuracy, by incrementing the amount of regularization being applied to the network?

In other words, is it correct to assume that there exists an specific setting for my regularization hyperparameters, that would allow to "almost completely" reduce that 21% gap, to something like 1% or 2% gap? At the cost of reducing training accuracy, getting for instance 79%-77% train and test accuracies.

I've found this other simmilar issue validation/training accuracy and overfitting, however I don't find any of the answers accurate enough to the particular question.

Answer 1

The accuracy gap between train and test data is caused by fitting:

1. noises in train data (hence over-fit)
2. features exist only in train data

The regularisation can control fitting to noises.

But for the second point, you may need a process like feature selection.