I have a simple network for classifying MNIST digits using Fully Connected layers. However I cannot explain why a hidden layer without activation makes the network behave randomly. There are three different network setups, to explain the issue.
1. Hidden Layer with Activation (94%)
Essentially, it is:
x = Input()
h = FullyConnected(x, units=100, activation=ReLU)
logits = FullyConnected(h, units=10, activation=None)
output = Softmax(logits)
The above network achieves 94% accuracy.
2. Hidden Layer without Activation (9.8%)
x = Input()
h = FullyConnected(x, units=100, activation=None)
logits = FullyConnected(h, units=10, activation=None)
output = Softmax(logits)
If I remove the activation, it gets an accuracy of 9.8%, which is close to random.
I understand that the activation provides the non-linerarity required for the hidden layer to be meaningful. And if I remove the activation for the hidden layer, it is equivalent to having just the final layer.
3. No Hidden Layer (90%)
However, a network with just the final layer achieves 90% accuracy.
x = Input()
logits = FullyConnected(x, units=10, activation=None)
output = Softmax(logits)
I am not sure why the second setup is worse than the third. If the hidden layer is redundant, I should be seeing a performance closer to 90%. It seems that the hidden layer without activation is preventing the network from learning.