Multilayer perceptron for binary classification: threshold learning

Question

In a basic contest, the MLP loss function (cross entropy) uses as value for the label ŷ:

• +1 if the net output is greater or equal to 0.5
• -1 otherwise

Where the net output is a value in [0,1] obtained from the logistic function:

g(z) = 1 / (1 + e^(-z))

The z value is obtained from the net itself (combination from layers + neurons + non linear activation functions). Details here

Now, the error minimization passes through these assumptions, and so the weights are changed consequently.

My questions are:

• Make it sense to learn, in the validation set, a different threshold (from 0.5) that will reduce better the error, even if in the process of learning itself the threshold is set to 0.5?

• Is the threshold value an hyper-parameter(learned in the validation phase) or a net parameter (learned in the training phase)?

Bonus question:

For the same reason, why can I say that the output g(z) from the net, could be interpreted as a probability outcome?

Answer 1

There is a small misinterpretation at your statement: The cross entropy is given by the following formula:

$$ L = -\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{m}y_{ij}\log(p_{ij})$$

Where $n$ is the number of predictions and $m$ is the number of classes. I could even write the loss function using the following expression:

$$ L = - \frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{m}y_{ij}\log\Big(\underbrace{\frac{e^{z_i}}{\sum_{j}e^{z_j} } }_\text{Softmax}\Big) $$

The output is not $+1$ or $-1$. Basically, you try to penalize not giving high probability to the right class for every prediction you make. If for the third prediction the right class is Class 2 and your MLP calculates that $p_{3,2} = 1$ then $log(p_{3,2}) = 0$ and your loss function doesn't increase. On the other hand if the right class for the 8th prediction is class 3 but your network after the softmax layer says that $p_{8,3} = 0.002$ then the term $log(p_{8,3})$ will be quite big and your loss function will be increased.

Usually lowering the threshold takes place when you care a lot about the metric called recall. For instance you develop an algorithm to detect terrorists in an airport and you want to find them all, even though some times you might identify normal people as terrorists. The metric that you are interested in this case is recall. On the other hand, increasing the threshold takes place when you care a lot about precision. This metric could be explained as: 'If I tell you that this guy is a terrorist he really is, but there are probably some terrorists that I haven't found them'. You can always change your threshold depending on what you want to achieve. You may also refer to Andrew Ng's course on that particular video.

As for the bonus question, indeed, you first predict the probability for each class before you calculate the cost function. In general you predict the probability, and then you take the probability for the right class to calculate the cost function (in our case cross entropy). You need this to minimize the loss function, but you can always have the probabilities for each class. I hope that I made things a bit more clear.