# Autoencoders can't learn meaningful features

I have 50,000 images such as these two:

They depict graphs of data. I wanted to extract features from these images so I used autoencoder code provided by Theano (deeplearning.net).

The problem is, these autoencoders don't seem to learn any features. I've tried RBM and it's the same.

MNIST dataset provides nice features but my data doesn't seem to yield any. I enclose examples below:

Filters created on MNIST:

Filters created by training on my data:

I've used many different permutations of hidden layer sizes and training epochs but the results are always the same.

Why doesn't it work? Why can't autoencoders extract features from these images?

EDIT:

For anyone that has a similar problem. The solution was really simple and the cause really dumb. I just forgot to rescale the pixel values from RGB encoding to float in range 0 - 1.

Rescaling values solved the problem.

Debugging neural networks usually involves tweaking hyperparameters, visualizing the learned filters, and plotting important metrics. Could you share what hyperparameters you've been using?

• What's your batch size?
• What's your learning rate?
• What type of autoencoder are you're using?
• Have you tried using a Denoising Autoencoder? (What corruption values have you tried?)
• How many hidden layers and of what size?
• What are the dimensions of your input images?

Analyzing the training logs is also useful. Plot a graph of your reconstruction loss (Y-axis) as a function of epoch (X-axis). Is your reconstruction loss converging or diverging?

Here's an example of an autoencoder for human gender classification that was diverging, was stopped after 1500 epochs, had hyperparameters tuned (in this case a reduction in the learning rate), and restarted with the same weights that were diverging and eventually converged.

Here's one that's converging: (we want this)

Vanilla "unconstrained"can run into a problem where they simply learn the identity mapping. That's one of the reasons why the community has created the Denoising, Sparse, and Contractive flavors.

Could you post a small subset of your data here? I'd be more than willing to show you the results from one of my autoencoders.

On a side note: you may want to ask yourself why you're using images of graphs in the first place when those graphs could easily be represented as a vector of data. I.e.,

[0, 13, 15, 11, 2, 9, 6, 5]


If you're able to reformulate the problem like above, you're essentially making the life of your auto-encoder easier. It doesn't first need to learn how to see images before it can try to learn the generating distribution.

Here are the filters from a 1000 hidden unit, single layer Denoising Autoencoder. Note that some of the filters are seemingly random. That's because I stopped training so early and the network didn't have time to learn those filters.

Here are the hyperparameters that I trained it with:

batch_size = 4
epochs = 100
pretrain_learning_rate = 0.01
finetune_learning_rate = 0.01
corruption_level = 0.2


I stopped pre-training after the 58th epoch because the filters were sufficiently good to post here. If I were you, I would train a full 3-layer Stacked Denoising Autoencoder with a 1000x1000x1000 architecture to start off.

Here are the results from the fine-tuning step:

validation error 24.15 percent
test error 24.15 percent


So at first look, it seems better than chance, however, when we look at the data breakdown between the two labels we see that it has the exact same percent (75.85% profitable and 24.15% unprofitable). So that means the network has learned to simply respond "profitable", regardless of the signal. I would probably train this for a longer time with a larger net to see what happens. Also, it looks like this data is generated from some kind of underlying financial dataset. I would recommend that you look into Recurrent Neural Networks after reformulating your problem into the vectors as described above. RNNs can help capture some of the temporal dependencies that is found in timeseries data like this. Hope this helps.

• My batch size is 20. The learning rate is set at 0.1. I've tried contractive and denoising autoencoders to no avail. I use one hidden layer. I've tried everything from 50 to 1000 neurons. I rescale the images to 25x25 - so they are even smaller than mnist. For what it's worth, I enclose all my data in this archive: Could you show me results of your autoencoders and this data? – b93dh44 Dec 31 '14 at 22:38
• The archive: mega.co.nz/… – b93dh44 Dec 31 '14 at 22:44
• OK, I've trained a network with that data. See the edit in the answer. – sabalaba Jan 1 '15 at 0:25
• That looks really interesting. Could you post your code? I'd really apprecieate it. I think I might have some bug in my code that prevents the network from training. I noticed you have fine-tuning stage? I think my autoencoder only pretrains without any finetuning. Could you show me your solution, if you don't mind? – b93dh44 Jan 1 '15 at 0:46
• Can't post my code because it's my company's, but I can point you to this tutorial which has example code to make a Denoising Autoencoder: deeplearning.net/tutorial/dA.html. All of the hyperparameter advice will hold for that code. – sabalaba Jan 1 '15 at 1:13

I don't have enough rep to comment, so I will put this into in answer. I don't know exact reason, however:

1. The pattern in bottom left region looks similar to your second example, and pattern in right bottom corner seems very like to your first example, when inspected closely. The question is, how much variety is in your source data? If all 50 000 images are variations of same pattern, these 3 meaningful feature maps we see can be quite enough for autoencoder to explain and reconstruct all your data.

2. Second, you might want to look at reconstruction error and actual reconstructed images. How good results are? If reconstruction error is low, you have might have an overfit, perhaps due to resons described below (or maybe combination of these 3 patterns is just enough to describe all data in interested in). Otherwise, autoencoder just can't learn how to reconstruct your data and you need larger autoencoder or better training algorithm.

• 1. I think there is a lot of variety in this data. Each one of these 50,000 graphs is slightly different. I think it's impossible the autoencoder could reconstruct them all based on just 3 filters. 2. Any tips for estimating reconstruction error? The code I am using provides the error but it is a really big number that doesn't tell me much. Should I compare the reconstruction error to the size of input or number of hidden layers? – b93dh44 Dec 31 '14 at 22:45