How do I intentionally design an overfitting neural network? To have a neural network that performs perfectly on training set, but poorly on validation set, what am I supposed to do? To simplify, let's consider it a CIFAR-10 classification task.
For example, "no dropout" and "no regularization" would help, but "more layers" doesn't necessarily. I'm also wondering, do "batch size", choice of optimizer make any difference on overfitting?
 A: Memorization
For absolute overfitting, you want a network that is technically capable to memorize all the examples, but fundamentally not capable of generalization. I seem to recall a story about someone training a predictor of student performance that got great results in the first year but was an absolute failure in the next year, which turned out to be caused by using all columns from a table as features, including the column with the sequential number of the student, and the system simply managed to learn that e.g. student #42 always gets good grades and student #43 has poor performance, which worked fine until next year when some other student was #42.
For an initial proof of concept on CIFAR, you could do the following:

*

*Pick a subset of CIFAR samples for which the color of top left corner pixel happens to be different for every image, and use that subset as your training data.

*Build a network where the first layer picks out only the RGB values of the top left corner and ignores everything else, followed by a comparably wide fully connected layer or two until the final classification layer.

*Train your system - you should get 100% on training data, and near-random on test data.

After that, you can extend this to a horribly overfitting system for the full CIFAR:

*

*As before, filter the incoming data so that it's possible to identify each individual item in training data (so a single pixel won't be enough) but so that it's definitely impossible to solve the actual problem from that data. Perhaps the first ten pixels in the top row would be sufficient; perhaps something from metadata - e.g. the picture ID, as in the student performance scenario.

*Ensure that there's no regularization of any form, no convolutional structures that imply translational independence, just fully connected layer(s).

*Train until 100% training accuracy and weep at the uselessness of the system.

A: Generally speaking, if you train for a very large number of epochs, and if your network has enough capacity, the network will overfit.  So, to ensure overfitting: pick a network with a very high capacity, and then train for many many epochs.  Don't use regularization (e.g., dropout, weight decay, etc.).
Experiments have shown that if you train for long enough, networks can memorize all of the inputs in the training set and achieve 100% accuracy, but this doesn't imply it'll be accurate on a validation set.  One of the primary ways we avoid overfitting in most work today is by early stopping: we stop SGD after a limited number of epochs.  So, if you avoid stopping early, and use a large enough network, you should have no problem causing the network to overfit.
Do you want to really force lots of overfitting?  Then add additional samples to the training set, with randomly chosen labels.  Now choose a really large network, and train for a long time, long enough to get 100% accuracy on the training set.  The extra randomly-labelled samples is likely to further impede any generalization and cause the network to perform even worse on the validation set.
A: Here are some things that I think might help.

*

*If you are free to change the network architecture try using a large but shallower network. Layers help a network learn higher level features and by the last layer the features are abstract enough for the network to "make sense of them". By forcing training on a shallower network, you are essentially crippling the network of this ability to form a hierarchy of increasingly higher-level concepts and forcing it to rote learn the data (overfit it, that is to say) for the sake of minimizing the loss.

*If this is again something you would be interested in exploring, you can try data-starving the network. Give a large network just a handful of training examples and it will try to overfit it. Better yet, give it examples that have minimum variability -- examples that look pretty much the same.

*Do not use stochastic gradient decent. Stochasticity helps reduce overfitting. So, use full-batch training! If you want to use stochastic gradient decent, then design your minibatches to have minimum variability.

A: I like your question a lot.
People often talk about overfitting, but may be not too many people realized that intentionally design an overfitting model is not a trivial task! Especially with large amount of data.
In the past, the data size is often limited. For example, couple hundreds data points. Then it is easy to have some overfitted model.
However, in "modern machine learning", the training data can be huge, say million of images, if any model can overfit it, then that would be already a great achievement.
So my answer to your question is, not an easy task, unless you are cheating by reduce your sample size.
A: Just reduce the training set to a few or even 1 example.
It's a good, simple way to test your code for some obvious bugs.
Otherwise, no, there's no magical architecture that always overfits. This is "by design." Machine learning algorithms that overfit easily aren't normally useful.
A: According to the Open AI paper Deep Double Descent, you need to have just a large enough neural network for a given dataset. Presumably this makes the NN powerful enough to perfectly learn the training data, but small enough that you don't get the generalisation effect of a large network. The paper is empirical, so the reason why it works is not theretically understood...
As you can see in the graph, you start off with an undersized network that doesn't learn the data. You can increase the size until it performs well on the test set, but further increases in size lead to overfitting and worse performance on the test set. Finally very large neural nets enter a different regime where test error keeps decreasing with size. Note that training error (show in a different graph) decreases monotonically.

A: If you're given a lot of freedom in the algorithm design, you can do the following :

*

*train one huge but shallow (ad probably non-convolutional, you really want it very powerful but very stupid) neural network to memorize the training set perfectly, as suggested by @Peteris and @Wololo (his solution has converted me). This network should give you both the classification and a boolean indicating whether this image is in your training set or not.


*To train this first network, you'll actually need additional training data from the outside, to train the "not in training set" part.


*train the best convnet that you can to actually do your task properly (without overfitting).


*During inference/evaluation,

*

*use the 1st network to infer whether the image is in the training set or not.

*

*If it is, output the classification you have "learnt by heart" in the 1st network,

*Otherwise, use the 2nd network to get the least likely classification for the image





That way, with a large-enough 1st network, you should have 100% accuracy on the training data, and worse-than-random (often near-0%, depending on the task) on the test data, which is "better" than 100% vs random output.
A: If you have a network with two layers of modifiable weights you can form arbitrary convex decision regions, where the lowest level neurons divide the input space into half-spaces and the second layer of neurons performs an "AND" operation to determine whether you are in the right sides of the half-spaces defining the convex region.  In the diagram below you can form regions r1 and r2 this way.  If you add an extra later, you can form arbitrary concave or disjoint decision regions by combining the outputs of the sub-networks defining the convex sub-regions.  I think I got this proof from Philip Wasserman's book "Neural Computing: Theory and Practice" (1989).

Thus is you want to over-fit, use a neural network with three hidden layers of neurons, use a huge number of hidden layer neurons in each layer, minimise the number of training patterns (if allowed by the challenge), use a cross-entropy error metric and train using a global optimisation algorithm (e.g. simulated annealing).
This approach would allow you to make a neural network that had convex sub-regions that surround each training pattern of each class, and hence would have zero training set error and would have poor validation performance where the class distributions overlap.
Note that over-fitting is about over-optimising the model.  An over-parameterised model (more weights/hidden units than necessary) can still perform well if the "data mismatch" is not over-minimised (e.g. by applying regularisation or early stopping or being fortunate enough to land in a "good" local minimum).
