This is my first attempt at making a convolutional neural network, and I'm having trouble making it perform the easiest task. Even though each separate part of the algorithm seems to work as expected (after performing crude unit tests), the network always converges to a clearly suboptimal result.
Because of the requirements of the task I'm trying to achieve, the network doesn't downsample or maxpool. The input and output are both stacks of several image layers (all images in all stacks have the same width and height), where each pixel of the images is normalized in the range [-1, 1]. I use a hard tanh activation on all hidden layers, and a regular tanh activation on the last layer.
To experiment with the simplest example, I am trying to train the network to accept an input image and output that same image back. Specifically, I use images of faces and want the network to behave as an identity function filter. If I set the network to have just 1 feature at the input and output layers and no hidden layers, then manually set the weights of the only convolution kernel to zeros everywhere except a lone 1 in the center, the result is almost exactly the identity function I wanted.
But if I randomly initialize the weights and have the network learn through gradient descent, training seems to grind to a halt before finding the identity function, even with a large learning constant and no regularization. The network manages to produce a very ugly version of the input image full of artifacts, though it does look like it made an honest attempt to make the output look like the input. When I look at the weights, they look nothing like the zeros-everywhere-but-the-middle setup I manually tried earlier. The weights in each kernel don't even seem radially symmetric about the center. This problem is less noticeable with 0~1 hidden layers, but becomes unbearable even with just 3 hidden layers. The artifacts appear even if I repeatedly train the network on a single image and look at how well it does on that same training example.
Is this really caused by local minima, or have I made a mistake somewhere with the architecture or creation of the network, or with the math of the gradient descent?
If it really is a local minimum problem, then does that mean there's no hope for this setup to perform anything useful, as it can't even reliably behave as an identity function?
Should I try a different cost function, seeing that artifacts usually cover a small area of the image even though they are visually very jarring? I don't know what I could use instead though.
Or should I initialize weights to strongly prefer a near-identity function in the first place? (My true task involves a facial filter that preserves most details of the input image.) This sounds too arbitrary though, and I think it's possible to make it work without this trick anyway since people have done similar tasks with convolutional neural networks, like image upscaling, without giving the network a head start.
If anyone is interested, my code is here.
