0
$\begingroup$

I saw this equation in somebody's code which is an alternative approach to implementing the softmax in order to avoid underflow by division by large numbers.

softmax = e^(matrix - logaddexp(matrix)) = E^matrix / sumexp(matrix)

logsumexp = scipy.special.logsumexp(matrix, axis=-1, keepdims=True)
softmax = np.exp(matrix - logsumexp)

I understand that when you log equations that use division you would then subtract, i.e. log(1/2) = log(1) - log(2). However, in the implantation of the code above, shouldn't they also log the matrix in order to subract the logsumexp?

$\endgroup$
0

2 Answers 2

2
$\begingroup$

The idea of working in log-space to avoid underflow requires that the intermediate objects you use to track progress are themselves on the log-scale --- you only convert back to regular scale at the end of the computation. (If you use intermediate objects that are not on the log-scale then you introduce underflow at that point in the computation, which then defeats the whole purpose.) Consequently, I strongly suspect that the object matrix is using values that are already measured on the log-scale, and therefore you do not need to take a logarithm of the elements of this object in the code. I recommend you review the initial settings for this object to see if this is how things have been done.

$\endgroup$
1
$\begingroup$

matrix is on the log scale. If matrix were not on the log scale, then you would only want to do log(sum(matrix)) not logsumexp(matrix).

To see why, write down the steps and work through the algebra. It should be quite apparent if you do the manipulations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.