1
$\begingroup$

I'm currently playing with auto-encoders, so the question is more about research than practical implementation. I know that if I reduce capacity of auto-encoder by making hidden layer smaller, I'll get some kind of factorization of the input (adding noise is more effective, but let's forget about that for a while).

However I also read that, given enough neurons on hidden layer auto-encoder will just learn to repeat input, so I wanted to check that worst case (to see how it actually does it) by building a small simple MLP:

  • no noise at the input
  • number of layers: 3 (inputN=4, contextN=4, outputN=4)
  • activation: ReLU
  • weight init: XAVIER
  • input/output number of features: 4
  • hidden layer size: 4
  • regularization: L2 (weight decay), tried no-regularization
  • SGD with back-propagation, no momentum
  • learning rate: 0.01 (tried 0.1, 0.001, 0.0001)
  • library: deeplearning4j

If I pass a single (or several) sample over and over - I can see that only some of output values repeat input, others are zeros. So, the output is becoming sparse, meaning that for let's say [0.1, 0.3, 0.7, 0.2] - I get something like [0.1, 0.0, 0.7, 0.0]. Running more iterations doesn't improve it after some point.

I believe the sparsity of result is due to gradient vanishing, probably because some of the initial weights were set close or equal to zero by XAVIER. I tried other random initializations (including normal distribution), but it didn't help; Of course, I could try to set weights as they should be (and I did - it worked), but it wouldn't show anything interesting.

So, is there any example of multi-layer auto-encoder that just repeats its input completely? Was my guess about weight initialization right or is there any another underlying reason? Does varying samples and order play any significant role in this particular case?

P.S. If this problem is not typical for this architecture - I could provide some reproducible example (in case there is a problem with my code or underlying library).

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ An autoencoder learns a representation of the dataset, not single input neurons. If an autoencoder repeats it's input perfectly we can assume it overfit somewhere. You can make the network overfit by giving it a very small dataset. Or you can make the latent vector bigger than the input vector. However, if the latent vector is smaller the network cannot reproduce it's inputs perfectly for every input possible, but it can for a subset of the input space, if that subset is small enough. $\endgroup$
    – Bloc97
    Commented Apr 11, 2018 at 13:51

1 Answer 1

Reset to default
2
$\begingroup$

Why not start with one layer, and get the output equals the input for one layer? Then, by induction, you can just stack those one on top of the other, and just pipeline through your trained identity layers.

Note that if you're using ReLU, then if you have any negative values in your input/output, then ReLU will not be able to do the identity operation you seek. So you'd probably want to make sure your input/output is strictly positive.

Gradient vanishing refers to a specific effect of backpropagated gradients being attenuated by passage through an activation function, like tanh or sigmoid. ReLU doesnt suffer from vanishing gradients too much, since the gradient is 1, for positive domain input.

So:

  • check your inputs/outputs are strictly positive
  • start with a single layer

Edit: based on your new information that it works with a single layer, but not with multiple layers, I reckon that ReLU is blocking your gradient backprop, probably because it has zero gradient for much of its domain. Therefore, you might try using an activation function that has non-zero gradient almost everywhere, eg ELU or leaky ReLU. On the whole, I'd try leaky ReLU first, because:

  • it is technically an activation function (cf not using an activation function at all)
  • its piecewise linear (cf ELU)
  • it has gradient almost everywhere, so gradients should backprop ok
Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the reply. 1) my values are strictly positive to avoid ReLU "saturation" as you could see from example 2) basically my problem is that I can do it with a single layer easily (because of no back-propagation there) - I want to see same behaviour for back-propagated layers, not for stacked ones... or at least get some explanation why it doesn't work with backprop. $\endgroup$
    – dk14
    Commented Mar 25, 2017 at 9:32
  • $\begingroup$ Hmmm. Interesting that works for one layer but not for multiple. Maybe whats happening is that ReLU is blocking your gradient backprop. I'd be tempted to try 1. using an activaiton function that has non-zero gradient almost everywhere, eg ELU, or leaky ReLU; and/or 2. adding dropout (to cause the neurons to be turned on/off randomly, albeit the noise will mean the network takes longer to converge). $\endgroup$
    – Hugh Perkins
    Commented Mar 25, 2017 at 9:34
  • $\begingroup$ (I guess leaky relu might do what you want: piecewise linear, so gradients wont vanish horribly, has gradient almost everywhere, is technically an activation function (cf just not having an activation function at all...), and is quite commonly used) $\endgroup$
    – Hugh Perkins
    Commented Mar 25, 2017 at 9:37
  • $\begingroup$ after applying LeakyRELU I've got rid of zeros but they were replaced with small negative values [0.6623673,0.12019485,0.13492113,0.57845426] ==> [-5.88048E-4,0.12378041,-0.00475002,-0.0019856603] $\endgroup$
    – dk14
    Commented Mar 25, 2017 at 9:52
  • $\begingroup$ actually, maybe there is some problem with my new configuration/code. I tried to use only one layer and got same results. I think I should re-check it again... $\endgroup$
    – dk14
    Commented Mar 25, 2017 at 10:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.