22
$\begingroup$

To have a neural network that performs perfectly on training set, but poorly on validation set, what am I supposed to do? To simplify, let's consider it a CIFAR-10 classification task.

For example, "no dropout" and "no regularization" would help, but "more layers" doesn't necessarily. I'm also wondering, do "batch size", choice of optimizer make any difference on overfitting?

$\endgroup$
  • 2
    $\begingroup$ Nothing "guarantees" overfitting. If there was something like this, we would simply be not using it when building the neural networks. $\endgroup$ – Tim Jun 30 at 9:00
  • 3
    $\begingroup$ @Tim: wouldn't just adding massive amounts of totally random data do the trick? $\endgroup$ – Stephan Kolassa Jun 30 at 9:17
  • 4
    $\begingroup$ Add a lot of completely random features to your net. Unless you prune/regularize, your net will latch on the spurious correlations and do better and better in training. And worse in testing/validation. You can even overfit on the test set, it's just a question of sifting through enough random data. See here. $\endgroup$ – Stephan Kolassa Jun 30 at 9:21
  • 2
    $\begingroup$ @StephanKolassa my experience says that simply adding more layers/channels doesn't usually improve training performance. $\endgroup$ – Rahn Jun 30 at 9:25
  • 3
    $\begingroup$ @DikranMarsupial: yes, as I wrote, "completely random features". $\endgroup$ – Stephan Kolassa Jun 30 at 9:46
10
$\begingroup$

If you have a network with two layers of modifiable weights you can form arbitrary convex decision regions, where the lowest level neurons divide the input space into half-spaces and the second layer of neurons performs an "AND" operation to determine whether you are in the right sides of the half-spaces defining the convex region. In the diagram below you can form regions r1 and r2 this way. If you add an extra later, you can form arbitrary concave or disjoint decision regions by combining the outputs of the sub-networks defining the convex sub-regions. I think I got this proof from Philip Wasserman's book "Neural Computing: Theory and Practice" (1989).

enter image description here

Thus is you want to over-fit, use a neural network with three hidden layers of neurons, use a huge number of hidden layer neurons in each layer, minimise the number of training patterns (if allowed by the challenge), use a cross-entropy error metric and train using a global optimisation algorithm (e.g. simulated annealing).

This approach would allow you to make a neural network that had convex sub-regions that surround each training pattern of each class, and hence would have zero training set error and would have poor validation performance where the class distributions overlap.

Note that over-fitting is about over-optimising the model. An over-parameterised model (more weights/hidden units than necessary) can still perform well if the "data mismatch" is not over-minimised (e.g. by applying regularisation or early stopping or being fortunate enough to land in a "good" local minimum).

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ The proof is either unnecessary (intuitively, it's possible to construct a bad network) or insufficient (so... how big does a network need to be, by this construction, to memorize CIFAR?) The actual suggestions, such as not randomizing the training data and using a batch optimizer (ie, batch size == epoch size) are good, but not sufficient. $\endgroup$ – Aleksandr Dubinsky Jun 30 at 20:24
4
$\begingroup$

Memorization

For absolute overfitting, you want a network that is technically capable to memorize all the examples, but fundamentally not capable of generalization. I seem to recall a story about someone training a predictor of student performance that got great results in the first year but was an absolute failure in the next year, which turned out to be caused by using all columns from a table as features, including the column with the sequential number of the student, and the system simply managed to learn that e.g. student #42 always gets good grades and student #43 has poor performance, which worked fine until next year when some other student was #42.

For an initial proof of concept on CIFAR, you could do the following:

  1. Pick a subset of CIFAR samples for which the color of top left corner pixel happens to be different for every image, and use that subset as your training data.
  2. Build a network where the first layer picks out only the RGB values of the top left corner and ignores everything else, followed by a comparably wide fully connected layer or two until the final classification layer.
  3. Train your system - you should get 100% on training data, and near-random on test data.

After that, you can extend this to a horribly overfitting system for the full CIFAR:

  1. As before, filter the incoming data so that it's possible to identify each individual item in training data (so a single pixel won't be enough) but so that it's definitely impossible to solve the actual problem from that data. Perhaps the first ten pixels in the top row would be sufficient; perhaps something from metadata - e.g. the picture ID, as in the student performance scenario.
  2. Ensure that there's no regularization of any form, no convolutional structures that imply translational independence, just fully connected layer(s).
  3. Train until 100% training accuracy and weep at the uselessness of the system.
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Overfitting is not when loss on train is much lower than loss on test (that's normal!). It is when the loss on the test set is much worse than it "should be," eg worse than assuming the prior. I'm not certain that this will happen. (You're not giving the net much useful data, so it obviously can't do well, but it might not do stupidly bad.) You have to try it. I suspect a non-tiny net will do just fine, and even make a prediction slightly better than the prior. Interesting experiment, regardless. $\endgroup$ – Aleksandr Dubinsky Jul 1 at 10:15
2
$\begingroup$

I like your question a lot.

People often talk about overfitting, but may be not too many people realized that intentionally design an overfitting model is not a trivial task! Especially with large amount of data.

In the past, the data size is often limited. For example, couple hundreds data points. Then it is easy to have some overfitted model.

However, in "modern machine learning", the training data can be huge, say million of images, if any model can overfit it, then that would be already a great achievement.

So my answer to your question is, not an easy task, unless you are cheating by reduce your sample size.

| cite | improve this answer | |
$\endgroup$
  • 5
    $\begingroup$ Not all problems that need solving are "big data" problems. We still need good methods for "small data" problems today. The difficulty in "big data" seems to be not inferential, but computational. For small data problems, handling the data is trivial, but the problem of inference is had. We should aim to have tools and skills for both kinds of problem, at least as a community if not as individuals. $\endgroup$ – Dikran Marsupial Jun 30 at 11:22
2
$\begingroup$

According to the Open AI paper Deep Double Descent, you need to have just a large enough neural network for a given dataset. Presumably this makes the NN powerful enough to perfectly learn the training data, but small enough that you don't get the generalisation effect of a large network. The paper is empirical, so the reason why it works is not theretically understood...

As you can see in the graph, you start off with an undersized network that doesn't learn the data. You can increase the size until it performs well on the test set, but further increases in size lead to overfitting and worse performance on the test set. Finally very large neural nets enter a different regime where test error keeps decreasing with size. Note that training error (show in a different graph) decreases monotonically.

test error by network size

| cite | improve this answer | |
New contributor
csiz is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
  • 1
    $\begingroup$ The paper has nothing to do with the question. $\endgroup$ – Aleksandr Dubinsky Jun 30 at 19:58
  • 1
    $\begingroup$ Also, it's not a very significant or interesting paper. The talked-about "phenomenon" occurs at the point where the network is still doing poorly on the training set. (The question stipulates the network doing "perfectly" on the training set.) It doesn't talk about real overfitting, but some irrelevant observation that the authors hope might be explained by a theory of neural networks. $\endgroup$ – Aleksandr Dubinsky Jun 30 at 20:07
  • 1
    $\begingroup$ @AleksandrDubinsky I agree that it's not terribly interesting, but it does seem relevant. I take it as a general phenomenon that intermediately sized networks perform worse on the test set than expected from the training error. Thus a rule of thumb for OP's question is to slightly increase the capacity of whatever small-ish neural net he finds that performs well. The train error does approach 0, though it's not quite there at the critical point. For this I'd challenge OP's assumption of what over-fitting means. Perhaps he wants to maximise test error for train error < threshold. $\endgroup$ – csiz Jul 1 at 0:00
  • 1
    $\begingroup$ The more relevant bit is the comparison to classical statistics, where the answer would simply be to increase the number of parameters. The paper shows it's not as simple for modern neural nets. $\endgroup$ – csiz Jul 1 at 0:08
2
$\begingroup$

Generally speaking, if you train for a very large number of epochs, and if your network has enough capacity, the network will overfit. So, to ensure overfitting: pick a network with a very high capacity, and then train for many many epochs. Don't use regularization (e.g., dropout, weight decay, etc.).

Experiments have shown that if you train for long enough, networks can memorize all of the inputs in the training set and achieve 100% accuracy, but this doesn't imply it'll be accurate on a validation set. One of the primary ways we avoid overfitting in most work today is by early stopping: we stop SGD after a limited number of epochs. So, if you avoid stopping early, and use a large enough network, you should have no problem causing the network to overfit.

Do you want to really force lots of overfitting? Then add additional samples to the training set, with randomly chosen labels. Now choose a really large network, and train for a long time, long enough to get 100% accuracy on the training set. The extra randomly-labelled samples is likely to further impede any generalization and cause the network to perform even worse on the validation set.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ "if you train for long enough, networks can memorize all of the inputs in the training set and achieve 100% accuracy, but this doesn't imply it'll be accurate on a validation set." That also doesn't imply it won't be accurate on the validation set. $\endgroup$ – Aleksandr Dubinsky Jul 1 at 10:18
  • $\begingroup$ @AleksandrDubinsky, I know. One can make the same comment about every answer here. See the last paragraph of my answer for what one can do about that. There are no mathematical guarantees in this realm, but I expect this is likely to work. $\endgroup$ – D.W. Jul 1 at 16:04
2
$\begingroup$

Here are some things that I think might help.

  1. If you are free to change the network architecture try using a large but shallower network. Layers help a network learn higher level features and by the last layer the features are abstract enough for the network to "make sense of them". By forcing training on a shallower network, you are essentially crippling the network of this ability to form a hierarchy of increasingly higher-level concepts and forcing it to rote learn the data (overfit it, that is to say) for the sake of minimizing the loss.
  2. If this is again something you would be interested in exploring, you can try data-starving the network. Give a large network just a handful of training examples and it will try to overfit it. Better yet, give it examples that have minimum variability -- examples that look pretty much the same.
  3. Do not use stochastic gradient decent. Stochasticity helps reduce overfitting. So, use full-batch training! If you want to use stochastic gradient decent, then design your minibatches to have minimum variability.
| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Just reduce the training set to a few or even 1 example.

It's a good, simple way to test your code for some obvious bugs.

Otherwise, no, there's no magical architecture that always overfits. This is "by design." Machine learning algorithms that overfit easily aren't normally useful.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

If you're given a lot of freedom in the algorithm design, you can do the following :

  • train one huge but shallow (ad probably non-convolutional, you really want it very powerful but very stupid) neural network to memorize the training set perfectly, as suggested by @Peteris and @Wololo (his solution has converted me). This network should give you both the classification and a boolean indicating whether this image is in your training set or not.

  • To train this first network, you'll actually need additional training data from the outside, to train the "not in training set" part.

  • train the best convnet that you can to actually do your task properly (without overfitting).

  • During inference/evaluation,

    • use the 1st network to infer whether the image is in the training set or not.
      • If it is, output the classification you have "learnt by heart" in the 1st network,
      • Otherwise, use the 2nd network to get the least likely classification for the image

That way, with a large-enough 1st network, you should have 100% accuracy on the training data, and worse-than-random (often near-0%, depending on the task) on the test data, which is "better" than 100% vs random output.

| cite | improve this answer | |
New contributor
gdelab is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.