Lets say I have 3 neurons in the last layer of my neural network and I am using mean squared error as a loss function. The desired output of my neural network is a vector:
If an activation function of those neurons is logistic sigmoid, they produce an output vector with a values between 0 and 1, for example:
[0.05, 0.80, 0.15].
So, I encode false as 0 and true as 1, and I can calculate the loss like this:
$$ (0 - 0.05)^2 + (1 - 0.80)^2 + (0 - 0.15)^2 = 0.065 $$
Now, let's say an activation function of the last layer of my neural network is a hyperbolic tangent, so it produces an output vector between -1 and 1:
[-0.95, 0.85, -0.75], so I encode
false as -1 and
true as 1 and my calculations for the loss look like this:
$$ (-1 + 0.95)^2 + (1 - 0.85)^2 + (-1 + 0.75)^2 = 0.0875 $$
That's all make sense for me, the question I have is how do I encode true and false values if the output of the last layer of my network does not have upper or lower bound ? i.e. if I am using ReLU as an activation function ?