I am trying to train a MLP using some training data in which the inputs are 32x32 matrices and the final output is a scalar.

Case 1: I build a network in which the first two hidden layers have the identity activation function i.e. the output from the 2nd layer is simply a linear transformation of the inputs. I trained my network and got decent results.

Case 2: I replaced the first two hidden layers from case 1 with just a single hidden layer with identity activation such that the dimensionality of the output from the 1st hidden layer in the 2nd case is the same as the dimensionality of the output from the first two hidden layers in case 1.

I found that generalization error was much larger in the case 2 as opposed to case 1.

Can anyone explain why this happens?


Did I get it correctly, that in the first case you use two layers with identity activation function and in the second case you use only one layer with the same activation function?

If yes, the question is, why do you need several such layers. They should be the same as you can reduce linear combination of linear combinations to just one linear combination:

$\sum_{j=1}^mw_j^{(2)}\sum_{i=1}^kw_i^{(1)}x_i = w_1^{(2)}\cdot(w_1^{(1)}x_1 + ... + w_k^{(1)}x_k) + ... + w_m^{(2)}\cdot(w_1^{(1)}x_1 + ... + w_k^{(1)}x_k) = (w_1^{(2)}w_1^{(1)}+...+w_m^{(2)}w_1^{(1)})\cdot x_1 +...+ (w_1^{(2)}w_k^{(1)}+...+w_m^{(2)}w_k^{(1)})\cdot x_k = \sum_{i=1}^kw_i^{(3)}x_i$

As for your concern, do you train them in the same way? Try to play with the number of epochs for, example. If you have further questions, please add more information.

  • $\begingroup$ Thank you ! I was able to get identical results eventually from both cases. However the hyperparameters I used in case 1 and case 2 are not identical. Any idea as to why this might be the case? $\endgroup$ Jul 7 '16 at 11:10
  • $\begingroup$ In two cases you have to train the different number of parameters (when you use the one layer - the number of weights you want to tune is fewer). $\endgroup$
    – yobibyte
    Jul 7 '16 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.