How can an autoencoder unroll the swiss roll?

I'm trying to unroll the swiss roll (from 3D to 2D) using an autoencoder, but it keeps getting stuck in local optima: the swiss roll ends up squashed rather than unrolled. It's no better than using PCA, basically.

If I just have one hidden layer with 2 units, the autoencoder just projects the swiss roll down to 2D, which squashes it (like PCA), and the decoder can't recover the unrolled version. So I tried to stack 3 hidden layers like so:

ouputs    (3 units)    \
|                     | decoder
hidden 3  (5 units)     |
|                    /
hidden 2  (2 units)    \
|                     | encoder
hidden 1  (5 units)     |
|                    /
inputs    (3D)

But for some reason this does not work any better. Since I'm pretty sure the weights could be set manually to make it work fine, I guess this means that training is stuck in a local minimum.

I have tried tying the weights ($W_{out} = {W_1}^T$, and $W_3 = {W_2}^T$) and adding $\ell_2$ regularization, but to no avail.

Any idea on what I should try next?

Edit

• I'm using the ELU activation function in all hidden layers, and no activation function in the output layer.
• I've successfully used this same architecture (with more neurons) to compress MNIST, so it works fine with more dimensions.
• You could start with the 2D case, which might allow easier visualization? (like here) Sep 29, 2016 at 15:02
• @GeoMatt22 Wow, what a cool link. Sep 29, 2016 at 15:15
• @amoeba Good point, I updated my question. I'm using ELU for all hidden layers (I tried ReLU as well) and no activation function for the output layer. Sep 29, 2016 at 15:34
• I believe the image recognition problems where NNs have had most success are actually quite linear (github.com/ducha-aiki/caffenet-benchmark/blob/master/… the linear model has 39% accuracy vs 52% for best non linear activation). Maybe you could just plot the error curve in a few random directions for MNIST vs swiss roll to see whether that highlights 'local minima'. Sep 29, 2016 at 17:25
• @amoeba -agreed it is nonlinear through max pooling but this is fixed. In the associated paper (fig 1)- the best net is ~72% using VGGNet and 227x227 input image. The network investigated uses a smaller input image (128x128) and fewer parameters allowing each training experiment to be done in a day vs a month. paper is linked from here github.com/ducha-aiki/caffenet-benchmark Oct 1, 2016 at 8:38