Effect on a Hidden Layer without Activation

Question

I have a simple network for classifying MNIST digits using Fully Connected layers. However I cannot explain why a hidden layer without activation makes the network behave randomly. There are three different network setups, to explain the issue.

1. Hidden Layer with Activation (94%)

Essentially, it is:

x = Input()
h = FullyConnected(x, units=100, activation=ReLU)
logits = FullyConnected(h, units=10, activation=None)
output = Softmax(logits)

The above network achieves 94% accuracy.

2. Hidden Layer without Activation (9.8%)

x = Input()
h = FullyConnected(x, units=100, activation=None)
logits = FullyConnected(h, units=10, activation=None)
output = Softmax(logits)

If I remove the activation, it gets an accuracy of 9.8%, which is close to random.

I understand that the activation provides the non-linerarity required for the hidden layer to be meaningful. And if I remove the activation for the hidden layer, it is equivalent to having just the final layer.

3. No Hidden Layer (90%)

However, a network with just the final layer achieves 90% accuracy.

x = Input()
logits = FullyConnected(x, units=10, activation=None)
output = Softmax(logits)

I am not sure why the second setup is worse than the third. If the hidden layer is redundant, I should be seeing a performance closer to 90%. It seems that the hidden layer without activation is preventing the network from learning.

Answer 1

The second and third options are equivalent in capacity. Normally, with two hidden layers, you have the following output from your network, where subscripts represent which layer matrices/functions belong to: $f_2(W_2\ f_1(W_1x+b_1)+b_2)$. In no-activation case, $f_1$ is unity so, this becomes $f_2(W_2W_1x+W_2b_1+b_2)$, which is nothing but $f(Wx+b)$, as in the third case. So, no matter how many layers you add, you won't get any extra capability. The result can be worse as you reported because in the second case you have more variables, i.e. extra complexity, which means more local minima, longer training times etc.