This is my first attempt at making a convolutional neural network, and I'm having trouble making it perform the easiest task. Even though each separate part of the algorithm seems to work as expected (after performing crude unit tests), the network always converges to a clearly suboptimal result.

Because of the requirements of the task I'm trying to achieve, the network doesn't downsample or maxpool. The input and output are both stacks of several image layers (all images in all stacks have the same width and height), where each pixel of the images is normalized in the range [-1, 1]. I use a hard tanh activation on all hidden layers, and a regular tanh activation on the last layer.

To experiment with the simplest example, I am trying to train the network to accept an input image and output that same image back. Specifically, I use images of faces and want the network to behave as an identity function filter. If I set the network to have just 1 feature at the input and output layers and no hidden layers, then manually set the weights of the only convolution kernel to zeros everywhere except a lone 1 in the center, the result is almost exactly the identity function I wanted.

But if I randomly initialize the weights and have the network learn through gradient descent, training seems to grind to a halt before finding the identity function, even with a large learning constant and no regularization. The network manages to produce a very ugly version of the input image full of artifacts, though it does look like it made an honest attempt to make the output look like the input. When I look at the weights, they look nothing like the zeros-everywhere-but-the-middle setup I manually tried earlier. The weights in each kernel don't even seem radially symmetric about the center. This problem is less noticeable with 0~1 hidden layers, but becomes unbearable even with just 3 hidden layers. The artifacts appear even if I repeatedly train the network on a single image and look at how well it does on that same training example.

Is this really caused by local minima, or have I made a mistake somewhere with the architecture or creation of the network, or with the math of the gradient descent?

If it really is a local minimum problem, then does that mean there's no hope for this setup to perform anything useful, as it can't even reliably behave as an identity function?

Should I try a different cost function, seeing that artifacts usually cover a small area of the image even though they are visually very jarring? I don't know what I could use instead though.

Or should I initialize weights to strongly prefer a near-identity function in the first place? (My true task involves a facial filter that preserves most details of the input image.) This sounds too arbitrary though, and I think it's possible to make it work without this trick anyway since people have done similar tasks with convolutional neural networks, like image upscaling, without giving the network a head start.

If anyone is interested, my code is here.

  • $\begingroup$ Asking for code check is off topic here. You could try on the Code Review SE site. This also may be a viable machine learning question for this site. You might decide which you prefer, & if you want ML advice, just edit to clarify that. $\endgroup$ Jul 5, 2016 at 3:36
  • 1
    $\begingroup$ I think it's more a machine learning question, and that maybe this behavior is something characteristic of convolutional neural networks. Or it might be an issue with weight initialization, initially biased behaviors of these networks, insufficient or too much complexity, or any number of things - all machine learning issues rather than programming mistakes. I thought it could be hard to describe every last detail of my network without including the code. Should I remove the link anyway? $\endgroup$
    – Jerry Ding
    Jul 5, 2016 at 3:59
  • $\begingroup$ No, I don't think the mere existence of the link should be problematic. Thanks for clarifying. $\endgroup$ Jul 5, 2016 at 11:37
  • $\begingroup$ @JerryDing The link to the code would be great, as it would allow others to understand your model in more detail. Did you find a solution by now? $\endgroup$ Jan 13, 2017 at 9:54


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.