I'm trying to unroll the swiss roll (from 3D to 2D) using an autoencoder, but it keeps getting stuck in local optima: the swiss roll ends up squashed rather than unrolled. It's no better than using PCA, basically.

If I just have one hidden layer with 2 units, the autoencoder just projects the swiss roll down to 2D, which squashes it (like PCA), and the decoder can't recover the unrolled version. So I tried to stack 3 hidden layers like so:

ouputs    (3 units)    \
  |                     | decoder
hidden 3  (5 units)     |
  |                    /
hidden 2  (2 units)    \
  |                     | encoder
hidden 1  (5 units)     |
  |                    /
inputs    (3D)

But for some reason this does not work any better. Since I'm pretty sure the weights could be set manually to make it work fine, I guess this means that training is stuck in a local minimum.

I have tried tying the weights ($W_{out} = {W_1}^T$, and $W_3 = {W_2}^T$) and adding $\ell_2$ regularization, but to no avail.

Any idea on what I should try next?


  • I'm using the ELU activation function in all hidden layers, and no activation function in the output layer.
  • I've successfully used this same architecture (with more neurons) to compress MNIST, so it works fine with more dimensions.
  • 4
    $\begingroup$ You could start with the 2D case, which might allow easier visualization? (like here) $\endgroup$
    – GeoMatt22
    Sep 29, 2016 at 15:02
  • 2
    $\begingroup$ @GeoMatt22 Wow, what a cool link. $\endgroup$
    – amoeba
    Sep 29, 2016 at 15:15
  • $\begingroup$ @amoeba Good point, I updated my question. I'm using ELU for all hidden layers (I tried ReLU as well) and no activation function for the output layer. $\endgroup$
    – MiniQuark
    Sep 29, 2016 at 15:34
  • 1
    $\begingroup$ I believe the image recognition problems where NNs have had most success are actually quite linear (github.com/ducha-aiki/caffenet-benchmark/blob/master/… the linear model has 39% accuracy vs 52% for best non linear activation). Maybe you could just plot the error curve in a few random directions for MNIST vs swiss roll to see whether that highlights 'local minima'. $\endgroup$
    – seanv507
    Sep 29, 2016 at 17:25
  • 1
    $\begingroup$ @amoeba -agreed it is nonlinear through max pooling but this is fixed. In the associated paper (fig 1)- the best net is ~72% using VGGNet and 227x227 input image. The network investigated uses a smaller input image (128x128) and fewer parameters allowing each training experiment to be done in a day vs a month. paper is linked from here github.com/ducha-aiki/caffenet-benchmark $\endgroup$
    – seanv507
    Oct 1, 2016 at 8:38


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.