Autoencoder doesn't work (can't learn features) I am completely new to machine learning and am playing around with the theanets package. What I am currently trying to do is to get an Autoencoder to reproduce a series of Gaussian distributions:
def gen_inputs(x=28, sigma=2.0):
    target = np.array([skimage.filters.gaussian_filter(i, sigma) 
                for i in ((np.eye(x*x)*2)).reshape(x, x, x*x).transpose()]).reshape(x*x, x*x)
    return target

I then attempted to use an Autoencoder (28*28, 9, 28*28) to train it. The parameters were as follows:


*

*learning_rate = 0.01

*input_noise = 0.01

*I used SGD with sigmoid activation function, along with linear output function. 


But my network couldn't reproduce the input. It seems to always converge to an average distribution of weights, resulting in random noise-like results. The loss function (MSE) converges as it should. Here are the results:

 A: (Primary author of theanets here.) As hinted in the comments on your question, this is actually a difficult learning problem! The network is, as indicated by the optimized loss value during training, learning the optimal filters for representing this set of input data as well as it can.
The important thing to think about here is that the weights in the network are being tuned to represent the entire space of inputs, not just one input. My guess is that you're expecting the network to learn one gaussian blob feature, but that's not how this works.
From the network's perspective, it's being asked to represent an input that is sampled from this pool of data arbitrarily. Which pixels in the next sample will be zero? Which ones will be nonzero? The network doesn't know, because the inputs tile the entire pixel space with zero and nonzero pixels. The best representation for a set of data that fills the space uniformly is a bunch of more or less uniformly-distributed small values, which is what you're seeing.
In comparison, try limiting your input data to a subset of the gaussian blobs. Let's put them all in a diagonal stripe of pixels, for instance:
import climate
import matplotlib.pyplot as plt
import numpy as np
import skimage.filters
import theanets

climate.enable_default_logging()

def gen_inputs(x=28, sigma=2.0):
    return np.array([
        skimage.filters.gaussian_filter(i, sigma).astype('f')
        for i in (np.eye(x*x)*2).reshape(x, x, x*x).transpose()
    ]).reshape(x*x, x*x)[10::27]


data = gen_inputs()

plt.imshow(data.mean(axis=0).reshape((28, 28)))
plt.show()

net = theanets.Autoencoder([784, 9, 784])
net.train(data, weight_l2=0.0001)

w = net.find('hid1', 'w').get_value().T
img = np.zeros((3 * 28, 3 * 28), float)
for r in range(3):
    for c in range(3):
        img[r*28:(r+1)*28, c*28:(c+1)*28] = w[r*3+c].reshape((28, 28))
plt.imshow(img)
plt.show()

Here's a plot of the mean data (the first imshow in the code):

And here's a plot of the learned features (the second imshow):

The features are responding to the mean of the entire dataset!
If you want to get the network to learn more "individual" features, it can be pretty tricky. Things you can play with:


*

*Increase the number of hidden units, as suggested in the comments.

*Try training with an L1 penalty on the hidden-unit activations (hidden_l1=0.5).

*Try forcing the weights themselves to be sparse (weight_l1=0.5).


Good luck!
