Lets say I have 3 neurons in the last layer of my neural network and I am using mean squared error as a loss function. The desired output of my neural network is a vector: [false,true,false]

If an activation function of those neurons is logistic sigmoid, they produce an output vector with a values between 0 and 1, for example: [0.05, 0.80, 0.15].

So, I encode false as 0 and true as 1, and I can calculate the loss like this:

$$ (0 - 0.05)^2 + (1 - 0.80)^2 + (0 - 0.15)^2 = 0.065 $$

Now, let's say an activation function of the last layer of my neural network is a hyperbolic tangent, so it produces an output vector between -1 and 1: [-0.95, 0.85, -0.75], so I encode false as -1 and true as 1 and my calculations for the loss look like this:

$$ (-1 + 0.95)^2 + (1 - 0.85)^2 + (-1 + 0.75)^2 = 0.0875 $$

That's all make sense for me, the question I have is how do I encode true and false values if the output of the last layer of my network does not have upper or lower bound ? i.e. if I am using ReLU as an activation function ?

  • 4
    $\begingroup$ Your task is binary classification, so why using MSE as a loss function in the first place, rather than binary cross-entropy? $\endgroup$ – DeltaIV Aug 6 '18 at 6:09
  • $\begingroup$ @DeltaIV thank you sir, I appreciate your suggestion to use different loss function, but the question is not about the choice of loss function for a given task, but rather how to encode true and false values for different activation functions. $\endgroup$ – koryakinp Aug 6 '18 at 21:45
  • 1
    $\begingroup$ I didn't suggest you to use binary cross-entropy. I asked why you are using MSE rather than binary cross-entropy. $\endgroup$ – DeltaIV Aug 6 '18 at 22:02
  • $\begingroup$ @DeltaIV, well, because I want. $\endgroup$ – koryakinp Aug 6 '18 at 22:14
  • 2
    $\begingroup$ It's worthwhile to consider that for logistic regression (equivalently a neural network with no hidden layers) using MSE instead of minimizing cross-entropy is non-convex. It seems like you will be borrowing trouble by going this route. $\endgroup$ – Sycorax Oct 10 '18 at 19:36

The standard1 way to perform classification with neural networks is to use sigmoid activation function and binary cross-entropy loss for single binary output, and linear activation followed by exponential normalization (softmax) and multinomial cross-entropy for one-hot binary output. There are good reasons why people use these.

If you want to use ReLU and mean squared error, you have to be prepared that things are probably not going to work optimally. But in principle it does not matter how you encode your values, your network should learn to predict them. It could be 0 and 1, or 0 and 34, or anything else. You cannot, however, expect the outputs of the network to be bounded within this range for arbitrary input. If that is among your requirements, use bounded activation function.

1 Understand as meaningful, accepted, working.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.