How are the subsequent layers trained exactly?

From what I understand, when the first layer is trained, image is split into small patches, those patches are then fed into an auto-encoder which then learns a sparse representation of the data - great! However, how are the larger representations learned (i.e. combinations of edges), since the input to the network is still the same - patches which capture just a single edge.

Do you have to "clone" your auto-encoders and feed, say four neighbouring patches and make next layer to learn on their representations (I am pretty sure this is not how it is done, but it might help show where my confusion is).


Subsequent layers learns from activation from previous layers. If you visualize activations you see that it is not like input ( there is no image there). What you can see from the first activation is likelihood of edges in different orientations. So the second layers takes presence of edges from the first layers and find more complex common structures out of it like combination of this activation. Combination of this activation means combination of edges so more complex structures like contours.

After learning, all data feedforward into first AE and activations from its output is used as input to next layer (next AE). Next AE learns more complex features and procedure goes on.


Inputs are randomly cropped mxm image patches collected from natural images. Input dimentions must be the same for after learning, however if you want to use those features for a convnet with mxm kernels, larger inputs are also possible. The reason for that the same features can be used to at different parts of the image. Say there is no reason certain edge only occurs down side of the image.

I've also completed UFLDL tutorial, if you are on Convnet section, the input can be larger. As far as I remember, Ng has used AE to pre-train convolution kernels but didn't fine tune them. However I don't remember larger input size at AE section, and it doesn't make sense to me now.

If this still don't answer your question, please indicate exact pages on the tutorial.

  • $\begingroup$ That much I do understand. However what is the input whilst training SAEs? I looked into UFLDL tutorial on SAEs and the input is the whole image when pre-training every layer. The confusion came from the fact that a lot of tutorials on Sparse Autoencoders, uses just patches of images, whilst in reality all image is used. $\endgroup$ May 12 '15 at 11:33
  • $\begingroup$ @spacemonkey I've updated my answer. $\endgroup$ May 12 '15 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.