Does the activation function of output layer differ during training and already trained network?

Question

I'm creating and OCR app, and so far it seams to work. It's quite similar to example from Coursera - Machine learning course.

Output layer of network has as many neurons as classes needed to recognize, 10 digits, 26 letter and alike. When the network is trained, it should basically output [1,0,0...0] for letter A, [0,1,0,...0] for letter B and so on.

In the description of hidden layers (during Coursera lessons), it is said that they use sigmoid function as activation function, and I'm not sure if output layer also uses sigmoid function.

So at the activation point, what differs when I train network, and when it is already trained?

Is it the same activation function except I should be turning largest output value into 1 and rest to 0, or there is something I'm missing here?

Explanation or links would be equally helpful. :)

Answer 1

JunJun yes that is correct - everything stays the same between training and test - you just take maximum to determine the classification. However, rather than a sigmoid on each class output, you might consider (training/testing) with softmax function on output layer: which is giving you 'true' probability in each class ( outputs sum to 1) Softmax

Answer 2

In a feed-forward neural network with a single hidden layer there's two activation functions. For some randomly selected batch sized $n$ of your training matrix, $X\in R^{n \cdot m}$, recall that the feed-forward step is given by $$A= F_1(W_1X + B_1)$$ $$\text{Pred} = F_2(W_2A + B_2)$$

The next step in training the model is back propagation.

You compute a bunch of $\delta$s, representing the errors, starting with the log loss.

$$\delta_2 = \frac{\sum(\text{Truth}\cdot \log(\text{Pred}) + (1 - \text{Truth}) \cdot \log(1 - \text{Pred})}{n}$$ $$\delta_1 = (W_2^T\delta_2)\cdot\nabla_wF_2(A)$$

Then update your weights, using some step-size $\alpha$.

$$W_2 := W_2 + \frac{\alpha}{n} A^T\delta_2$$ $$B_2 := B_2 + \frac{\alpha}{n}\delta_2$$ $$W_1 := W_1 + \frac{\alpha}{n} X^T\delta_1$$ $$B_1 := B_1 + \frac{\alpha}{n}\delta_1$$

This is repeated until $\delta_2$ is sufficiently small. Typically you monitor this on a held-out dataset.

Note that since you're using the sigmoid function there's a simple result for the calculation of gradient of $F$. $$\nabla_w F(x) = \nabla_w\frac{1}{1+e^{-wx}}\Rightarrow F(x) (1 - F(x))$$

Weight initialization of $W$s is on the interval $[0, \frac{1}{\sqrt{m}}]$ and $B$s as 0. There's a fancier way to initialize $W$, but it's probably not worth the effort with a simple network.

There is a boatload of room improvement over this simple model. Structurally some easy wins are convolutional inputs (given that this is an image recognition task) and rectified linear hidden units (which don't suffer from the vanishing gradient problem).

During training dropout, Nesterov's accelerated gradient and an $\ell_2$ norm can all make dramatic improvements.

If you don't want to implement all of this, I'd recommend looking into Lua Torch's gpu neural network library and Theano/Pylearn2, then you only need to implement the prediction method.

There's also a lot of interesting research on neural networks by Hinton and Bengio.