Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
To have a neural network that performs perfectly on training set, but poorly on validation set, what am I supposed to do? To simplify, let's consider it a CIFAR-10 classification task.
For example, "no dropout" and "no regularization" would help, but "more layers" doesn't necessarily. I'm also wondering, do "batch size", choice of optimizer make any difference on overfitting?
If you have a network with two layers of modifiable weights you can form arbitrary convex decision regions, where the lowest level neurons divide the input space into half-spaces and the second layer of neurons performs an "AND" operation to determine whether you are in the right sides of the half-spaces defining the convex region. In the diagram below you can form regions r1 and r2 this way. If you add an extra later, you can form arbitrary concave or disjoint decision regions by combining the outputs of the sub-networks defining the convex sub-regions. I think I got this proof from Philip Wasserman's book "Neural Computing: Theory and Practice" (1989).
Thus is you want to over-fit, use a neural network with three hidden layers of neurons, use a huge number of hidden layer neurons in each layer, minimise the number of training patterns (if allowed by the challenge), use a cross-entropy error metric and train using a global optimisation algorithm (e.g. simulated annealing).
This approach would allow you to make a neural network that had convex sub-regions that surround each training pattern of each class, and hence would have zero training set error and would have poor validation performance where the class distributions overlap.
Note that over-fitting is about over-optimising the model. An over-parameterised model (more weights/hidden units than necessary) can still perform well if the "data mismatch" is not over-minimised (e.g. by applying regularisation or early stopping or being fortunate enough to land in a "good" local minimum).
For absolute overfitting, you want a network that is technically capable to memorize all the examples, but fundamentally not capable of generalization. I seem to recall a story about someone training a predictor of student performance that got great results in the first year but was an absolute failure in the next year, which turned out to be caused by using all columns from a table as features, including the column with the sequential number of the student, and the system simply managed to learn that e.g. student #42 always gets good grades and student #43 has poor performance, which worked fine until next year when some other student was #42.
For an initial proof of concept on CIFAR, you could do the following:
After that, you can extend this to a horribly overfitting system for the full CIFAR:
Generally speaking, if you train for a very large number of epochs, and if your network has enough capacity, the network will overfit. So, to ensure overfitting: pick a network with a very high capacity, and then train for many many epochs. Don't use regularization (e.g., dropout, weight decay, etc.).
Experiments have shown that if you train for long enough, networks can memorize all of the inputs in the training set and achieve 100% accuracy, but this doesn't imply it'll be accurate on a validation set. One of the primary ways we avoid overfitting in most work today is by early stopping: we stop SGD after a limited number of epochs. So, if you avoid stopping early, and use a large enough network, you should have no problem causing the network to overfit.
Do you want to really force lots of overfitting? Then add additional samples to the training set, with randomly chosen labels. Now choose a really large network, and train for a long time, long enough to get 100% accuracy on the training set. The extra randomly-labelled samples is likely to further impede any generalization and cause the network to perform even worse on the validation set.
I like your question a lot.
People often talk about overfitting, but may be not too many people realized that intentionally design an overfitting model is not a trivial task! Especially with large amount of data.
In the past, the data size is often limited. For example, couple hundreds data points. Then it is easy to have some overfitted model.
However, in "modern machine learning", the training data can be huge, say million of images, if any model can overfit it, then that would be already a great achievement.
So my answer to your question is, not an easy task, unless you are cheating by reduce your sample size.
According to the Open AI paper Deep Double Descent, you need to have just a large enough neural network for a given dataset. Presumably this makes the NN powerful enough to perfectly learn the training data, but small enough that you don't get the generalisation effect of a large network. The paper is empirical, so the reason why it works is not theretically understood...
As you can see in the graph, you start off with an undersized network that doesn't learn the data. You can increase the size until it performs well on the test set, but further increases in size lead to overfitting and worse performance on the test set. Finally very large neural nets enter a different regime where test error keeps decreasing with size. Note that training error (show in a different graph) decreases monotonically.
Here are some things that I think might help.
Just reduce the training set to a few or even 1 example.
It's a good, simple way to test your code for some obvious bugs.
Otherwise, no, there's no magical architecture that always overfits. This is "by design." Machine learning algorithms that overfit easily aren't normally useful.
If you're given a lot of freedom in the algorithm design, you can do the following :
train one huge but shallow (ad probably non-convolutional, you really want it very powerful but very stupid) neural network to memorize the training set perfectly, as suggested by @Peteris and @Wololo (his solution has converted me). This network should give you both the classification and a boolean indicating whether this image is in your training set or not.
To train this first network, you'll actually need additional training data from the outside, to train the "not in training set" part.
train the best convnet that you can to actually do your task properly (without overfitting).
During inference/evaluation,
That way, with a large-enough 1st network, you should have 100% accuracy on the training data, and worse-than-random (often near-0%, depending on the task) on the test data, which is "better" than 100% vs random output.
