# MSE and different types of activation functions in NN

Lets say I have 3 neurons in the last layer of my neural network and I am using mean squared error as a loss function. The desired output of my neural network is a vector: [false,true,false]

If an activation function of those neurons is logistic sigmoid, they produce an output vector with a values between 0 and 1, for example: [0.05, 0.80, 0.15].

So, I encode false as 0 and true as 1, and I can calculate the loss like this:

$$(0 - 0.05)^2 + (1 - 0.80)^2 + (0 - 0.15)^2 = 0.065$$

Now, let's say an activation function of the last layer of my neural network is a hyperbolic tangent, so it produces an output vector between -1 and 1: [-0.95, 0.85, -0.75], so I encode false as -1 and true as 1 and my calculations for the loss look like this:

$$(-1 + 0.95)^2 + (1 - 0.85)^2 + (-1 + 0.75)^2 = 0.0875$$

That's all make sense for me, the question I have is how do I encode true and false values if the output of the last layer of my network does not have upper or lower bound ? i.e. if I am using ReLU as an activation function ?

• Your task is binary classification, so why using MSE as a loss function in the first place, rather than binary cross-entropy? – DeltaIV Aug 6 '18 at 6:09
• @DeltaIV thank you sir, I appreciate your suggestion to use different loss function, but the question is not about the choice of loss function for a given task, but rather how to encode true and false values for different activation functions. – koryakinp Aug 6 '18 at 21:45
• I didn't suggest you to use binary cross-entropy. I asked why you are using MSE rather than binary cross-entropy. – DeltaIV Aug 6 '18 at 22:02
• @DeltaIV, well, because I want. – koryakinp Aug 6 '18 at 22:14
• It's worthwhile to consider that for logistic regression (equivalently a neural network with no hidden layers) using MSE instead of minimizing cross-entropy is non-convex. It seems like you will be borrowing trouble by going this route. – Sycorax Oct 10 '18 at 19:36