2
$\begingroup$

I've implemented a denoising autoencoder using TensorFlow. The code is here, there is also a command line script to launch it. The code seems to work, the cross-validation error is decreasing every iteration, but the autoencoder doesn't seem to be learning good features (I'm using MNIST). This is an example of learned features: enter image description here

The parameters I used are the following:

--n_components 1000 --batch_size 25 --n_iter 100 --verbose 1 --learning_rate 0.01 --weight_images 0 --corr_type masking --corr_frac 0.5 --encode_valid --enc_act_func sigmoid --dec_act_func sigmoid --loss_func cross_entropy --opt momentum --momentum 0.9 --dropout 0.5

number of hidden units: 1000

batch_size: 25

epochs: 100

learning rate: 0.01

input corruption type and frac: masking 0.5 (set 50% of the pixels to zero)

encoder activation function: sigmoid

decoder activation function: sigmoid

loss function: cross entropy

optimizer: momentum, 0.9

encoder layer dropout probability: 0.5

The question is: what is a good choice of the hyperparameters for the MNIST dataset?

Improve this question
$\endgroup$
2
  • $\begingroup$ Can we ask for couple of more info? How do you initialize weights? Cause it seems your weight couldn't escape the symmetry. All look similar. What is the final loss value? $\endgroup$
    – ozgur
    Jul 22, 2016 at 19:34
  • $\begingroup$ Do you know specifically what features you should get? You have 1000 units trying to represent 10 classes for mnist, yes? So what makes you expect to get 1000 distinct features? Not all cost functions and models will interpretable features. Checkout other features learned e.g. from the dropout paper for MNIST on page 15 with and without dropout with 256 hidden units for a regular neural network. They have a different model but I would argue the features learned in that paper aren't super interpretable either. $\endgroup$
    – user27886
    Dec 13, 2016 at 14:45

2 Answers 2

Reset to default
2
$\begingroup$

Hyperparameter choice is something that can't really be answered: sure, there are some set of procedures that can be followed, but it's largely a case of hit and trial.

Single DA's can indeed extract meaningful features, however, most of the features in case of encoding dimensions 'L' (say) > input dimensions 'D' (i.e. Overcomplete learning) will end up being random noise. The reason for your autoencoder not learning meaningful features is because given the degree of freedom the autoencoder has in the encoding layer (i.e. L > D) it becomes quite easy for the autoencoder to learn an identity mapping of the input.

So to alleviate this problem, you have to put additional constraints in order to limit this degree of freedom. I believe you can try the following and see what the outcome is:

  1. The first and probably the easiest step would be to try and reduce the number of encoding layer nodes from 1000 to something little closer to the dimensions of the input, ie. 784. I would say 800 would be a good start. Visualize the features then and see if some features have improved.

  2. Apply additional regularization constraints, say l2 regularization on the weights (and if already doing that, increase the penalty term corresponding to l2) and other such penalization techniques.

  3. Tied weights. Use tied weights on the encoding layer and the decoding layer if not doing so already. ie. W_decoding = W_encoding.T. When not using tied weights, many times, either of the two layers learn larger, better weights (for the lack of words) and compensate for the poor weights learned by the other. By placing this constraint we force the autoencoder to learn a balanced set of weights. Also, it often results in improvement of training time as well as a pretty good limitation on the degree of freedom (the number of free, trainable parameters is halved!).

Give this a try. Might help.

Improve this answer
$\endgroup$
0
$\begingroup$

I am also just getting into DAs, however, one thing that I noticed that you might want to think about your input dimension versus the number of hidden units. The MNIST consists of 28x28 pixel images, that is an input dimension of 784 and you also have a corruption rate of 0.5 which seems quite high. 1000 hidden nodes might not be optimal in this if you want to construct relevant features without further stacking DAs?

Would have suited better as a comment but I don't have enough rep for this.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I am just getting started too. The idea is stacking DAs, but I would expect a single DA to extract meaningful features even without stacking and fine tuning. Am I wrong? However, I'll try with a lower number of hidden nodes, thanks! $\endgroup$
    – Blackecho
    Feb 9, 2016 at 11:44
  • $\begingroup$ Yeah, it should definitely be possible, but naturally it seems that a lower number of hidden units would easier find a lower dimensional representation of the data (abstract features). I am not sure how you initialize the weights, but this as I understand is very important to get convergence. You want them to be in the linear region of the activation function at the start to get the gradient descent going. $\endgroup$
    – johnblund
    Feb 9, 2016 at 13:16
  • $\begingroup$ I initialize the weights using a uniform random distribution between - k * sqrt(6 / (fan in + fan out)) and k * sqrt(6 / (fan in + fan out)) where I use k = 1 with the tanh activation function and k = 4 for the sigmoid. Thanks for making me notice that I forgot to include the weights initialization function in the code :) $\endgroup$
    – Blackecho
    Feb 10, 2016 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.