How can I overfit a fully-connected neural network to predict RGB values from (x,y) coordinates? The problem is the following:
Given a single 3-channel image (e.g. 200x150), I constructed a dataset where the features are the pairs of (x,y) coordinates and the targets are the (R,G,B) values. Each {(x,y) , (r,g,b)} is a training example. The aim is to overfit the training set (another way to see this is to be able to reconstruct the image pixel by pixel). 
I would like to achieve an almost perfect reconstruction, but even with 


*

*a neural network with 4 hidden layers

*ReLU activation function in each layer, except the output layer

*1.000.000 parameters

*normalizing features and targets between [0,1]

*training 300 epochs with rmsprop

*weights from a normal with mean 0 and std 0.05 and the biases at 0.


I can only achieve 0.005 mean squared error (normalized). 
How can I improve this performance? Do I need better preprocessing, network architecture, ecc, ...?
summary: The network is pretty useless, bu you can interpret the question this way:
How can I overfit a dataset with 200x150=30k training examples, each with 2 features (x,y) and 3 targets (r,g,b), With range(x) = [0,Width), range(y) = [0, Height) and range r,g,b = [0,255], using a fully-connected neural network?
 A: The SIREN paper ("Implicit Neural Representations with Periodic Activation Functions" --  NeurIPS 2020) earlier this year shows that ReLU networks are pretty bad at that task.
Instead, they propose to use the sine as activation function and show improved performance in a variety of tasks.
So, perhaps try exchanging ReLU for sine, that might solve your problem.
A: You have 30k $(x_i,y_i)$ inputs and 30k $(r_i,g_i,b_i)$ targets. $i$ goes from 1 to 30k.
Consider the following network. For each $((x,y), (r,g,b))$ pair you have 3 "linear neurons", each with 2 weights:
$r = \omega_{r,x} x + \omega_{r,y} y$
$g = \omega_{g,x} x + \omega_{g,y} y$
$b = \omega_{b,x} x + \omega_{b,y} y$
The complete set of neurons:
$r_i = \omega_{i,r,x} x + \omega_{i,r,y} y$
$g_i = \omega_{i,g,x} x + \omega_{i,g,y} y$
$b_i = \omega_{i,b,x} x + \omega_{i,b,y} y$
That's a total of 30k * 2 * 3 = 180k weights $\omega$. This will fit exactly.
EDIT: Each of those three systems has 3 equatins and 6 parameters, so they're underdetermined. An alternative which is exactly determined would be:
$r_i = \omega_{i,r} x + y$
$g_i = \omega_{i,g} x + y$
$b_i = \omega_{i,b} x + y$
This "neural network" now has $90k$ weights.
