4
$\begingroup$

The problem is the following: Given a single 3-channel image (e.g. 200x150), I constructed a dataset where the features are the pairs of (x,y) coordinates and the targets are the (R,G,B) values. Each {(x,y) , (r,g,b)} is a training example. The aim is to overfit the training set (another way to see this is to be able to reconstruct the image pixel by pixel).

I would like to achieve an almost perfect reconstruction, but even with

  • a neural network with 4 hidden layers
  • ReLU activation function in each layer, except the output layer
  • 1.000.000 parameters
  • normalizing features and targets between [0,1]
  • training 300 epochs with rmsprop
  • weights from a normal with mean 0 and std 0.05 and the biases at 0.

I can only achieve 0.005 mean squared error (normalized). How can I improve this performance? Do I need better preprocessing, network architecture, ecc, ...?

summary: The network is pretty useless, bu you can interpret the question this way: How can I overfit a dataset with 200x150=30k training examples, each with 2 features (x,y) and 3 targets (r,g,b), With range(x) = [0,Width), range(y) = [0, Height) and range r,g,b = [0,255], using a fully-connected neural network?

Improve this question
$\endgroup$
16
  • 3
    $\begingroup$ You are trying to reconstruct an image ... from itself?? With 90,001 parameters you can store it perfectly! $\endgroup$
    – whuber ♦
    Commented Mar 29, 2018 at 20:38
  • 2
    $\begingroup$ The information content of your image is only 200x150x3 Bytes=90,000 Bytes, which means that in the worst case you should need no more than 90K/2 = 45,000 single precision floating point parameters $\endgroup$
    – Aksakal
    Commented Mar 29, 2018 at 20:38
  • 1
    $\begingroup$ It seems interesting enough from the perspective of testing the capacity of a network / how capable SGD is of making use of that capacity. $\endgroup$
    – shimao
    Commented Mar 29, 2018 at 21:10
  • 1
    $\begingroup$ @oneloop You can forget that we are talking about an image. The underlying aim is to overfit a training set with 30k examples with (x,y) as features, and 3 (r,g,b) targets. This can be translated in being able to reconstruct the image, pixel by pixel, given its coordinates, but it is not the principal goal. $\endgroup$
    – Karl Alexius
    Commented Mar 30, 2018 at 10:09
  • 1
    $\begingroup$ By the way, I was able to push the error down to 0.002 MSE using a similar architecture as OP by decaying the learning rate a factor of 10 every 200 epochs from 1E-3 to 1E-7. Although it's not close to being perfect yet. $\endgroup$
    – shimao
    Commented Apr 1, 2018 at 13:51

3 Answers 3

Reset to default
2
$\begingroup$

The SIREN paper ("Implicit Neural Representations with Periodic Activation Functions" -- NeurIPS 2020) earlier this year shows that ReLU networks are pretty bad at that task.

Instead, they propose to use the sine as activation function and show improved performance in a variety of tasks.

So, perhaps try exchanging ReLU for sine, that might solve your problem.

Improve this answer
$\endgroup$
1
$\begingroup$

You have 30k $(x_i,y_i)$ inputs and 30k $(r_i,g_i,b_i)$ targets. $i$ goes from 1 to 30k.

Consider the following network. For each $((x,y), (r,g,b))$ pair you have 3 "linear neurons", each with 2 weights:

$r = \omega_{r,x} x + \omega_{r,y} y$

$g = \omega_{g,x} x + \omega_{g,y} y$

$b = \omega_{b,x} x + \omega_{b,y} y$

The complete set of neurons:

$r_i = \omega_{i,r,x} x + \omega_{i,r,y} y$

$g_i = \omega_{i,g,x} x + \omega_{i,g,y} y$

$b_i = \omega_{i,b,x} x + \omega_{i,b,y} y$

That's a total of 30k * 2 * 3 = 180k weights $\omega$. This will fit exactly.

EDIT: Each of those three systems has 3 equatins and 6 parameters, so they're underdetermined. An alternative which is exactly determined would be:

$r_i = \omega_{i,r} x + y$

$g_i = \omega_{i,g} x + y$

$b_i = \omega_{i,b} x + y$

This "neural network" now has $90k$ weights.

Improve this answer
$\endgroup$
9
  • $\begingroup$ This is not exactly what I am trying to do. My network would have 2 input neurons (a single pixel coordinate (x_i, y_i) ) and 3 output neurons (the RGB values for that pixel). $\endgroup$
    – Karl Alexius
    Commented Mar 30, 2018 at 10:22
  • $\begingroup$ You asked to overfit. With 2 neurons you can't overfit 30k parameters. $\endgroup$
    – oneloop
    Commented Mar 30, 2018 at 10:26
  • $\begingroup$ Also your OWN example says that you tried 4 hidden layers, and 1M parameters, so you're not trying just 2 neurons. $\endgroup$
    – oneloop
    Commented Mar 30, 2018 at 10:27
  • $\begingroup$ Right, in the comment above I just clarified the input and output dimensions (because it looked like you understood I was using 30k inputs and 30 outputs). So, I use 2 inputs neurons, 3 output neurons and 4 hidden layers with, say 500 neurons each. I also tried with 5 hidden layers, or with 1000 neurons per layer, without improvements w.r.t the 3-500-500-500-500-2 architecture. Hope it is clear now. $\endgroup$
    – Karl Alexius
    Commented Mar 30, 2018 at 10:54
  • 1
    $\begingroup$ @ThanksBye Ok, in that case I pose the following question to you: what makes you think that specific architecture will allow overfitting this specific problem? I mean, it might or might not, but can you prove either way? $\endgroup$
    – oneloop
    Commented Mar 30, 2018 at 11:20
0
$\begingroup$

This problem has been solved for the 3D case by NeRFs. The problem is that the network cannot easily tell the difference between position (0,0) and (0,1) - it doesn't have the input resolution to distinguish them.

To solve this, you need to spread the information across more inputs using a positional encoding - see "Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains" where they address your exact scenario.

More info: https://bmild.github.io/fourfeat/index.html

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.