A Deep Stacked Network (DSN), is a ensemble learner, which roughly works by training a single hidden layer neural network on the inputs and target outputs, then training another which takes an input the output of the previous layer(/s), as well as the input.

Due to a variation in the design of the neural network, it can be (and is) trained using It is trained using standard convex optimization, pseudo-inverse of the product.

Note: It is distinct from a Deep Neural Network and from a Deep Belief Network, and while inspired by, is not the same.

So I have implemented the basic algorithm (from 1), no RBM initialization, no fine tuning. It is fitting really tightly to the training data. Even with a single layer the squared difference is $<10^{-6}$. However it is over-fitting, badly. On the test set the classification accuracy (both with and without a softmax layer) is $<30\%$.

This is a difficult dataset, for sure. What techniques can reduce overfitting for this?

I suspect there are some standard techniques from convect optimization with pseudoinverse of product, that I am not aware of (I am well informed on the techniques for dealing with this in gradient descent situations).

So far I have tried:

  • Shrinking the hidden layer (Smaller hidden layer, less Representative capacity)
  • Increasing the variance in the random weight in the input weights ($W$).
  • $\begingroup$ try your implementation on some test dataset e.g. popular MINST, overfitting could be reduced by sparsness constrains $\endgroup$ – Qbik May 31 '15 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.