1
$\begingroup$

I have a simple network for classifying MNIST digits using Fully Connected layers. However I cannot explain why a hidden layer without activation makes the network behave randomly. There are three different network setups, to explain the issue.

1. Hidden Layer with Activation (94%)

Essentially, it is:

x = Input()
h = FullyConnected(x, units=100, activation=ReLU)
logits = FullyConnected(h, units=10, activation=None)
output = Softmax(logits)

The above network achieves 94% accuracy.

2. Hidden Layer without Activation (9.8%)

x = Input()
h = FullyConnected(x, units=100, activation=None)
logits = FullyConnected(h, units=10, activation=None)
output = Softmax(logits)

If I remove the activation, it gets an accuracy of 9.8%, which is close to random.

I understand that the activation provides the non-linerarity required for the hidden layer to be meaningful. And if I remove the activation for the hidden layer, it is equivalent to having just the final layer.

3. No Hidden Layer (90%)

However, a network with just the final layer achieves 90% accuracy.

x = Input()
logits = FullyConnected(x, units=10, activation=None)
output = Softmax(logits)

I am not sure why the second setup is worse than the third. If the hidden layer is redundant, I should be seeing a performance closer to 90%. It seems that the hidden layer without activation is preventing the network from learning.

$\endgroup$
2
$\begingroup$

The second and third options are equivalent in capacity. Normally, with two hidden layers, you have the following output from your network, where subscripts represent which layer matrices/functions belong to: $f_2(W_2\ f_1(W_1x+b_1)+b_2)$. In no-activation case, $f_1$ is unity so, this becomes $f_2(W_2W_1x+W_2b_1+b_2)$, which is nothing but $f(Wx+b)$, as in the third case. So, no matter how many layers you add, you won't get any extra capability. The result can be worse as you reported because in the second case you have more variables, i.e. extra complexity, which means more local minima, longer training times etc.

$\endgroup$
  • $\begingroup$ Yes, I understand that I would not get additional capacity. But over-parameterization should ideally not take me from 90% to 9.8%, which is basically random, right? $\endgroup$ – reddragon Dec 22 '18 at 16:13
  • 2
    $\begingroup$ It seems that it was indeed stuck in local minima, reducing the learning rate fixed it. Thanks to an offline suggestion by someone :-) $\endgroup$ – reddragon Dec 22 '18 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.