I am trying to train a MLP using some training data in which the inputs are 32x32 matrices and the final output is a scalar.
Case 1: I build a network in which the first two hidden layers have the identity activation function i.e. the output from the 2nd layer is simply a linear transformation of the inputs. I trained my network and got decent results.
Case 2: I replaced the first two hidden layers from case 1 with just a single hidden layer with identity activation such that the dimensionality of the output from the 1st hidden layer in the 2nd case is the same as the dimensionality of the output from the first two hidden layers in case 1.
I found that generalization error was much larger in the case 2 as opposed to case 1.
Can anyone explain why this happens?