using logsumexp in softmax

Question

I saw this equation in somebody's code which is an alternative approach to implementing the softmax in order to avoid underflow by division by large numbers.

softmax = e^(matrix - logaddexp(matrix)) = E^matrix / sumexp(matrix)

logsumexp = scipy.special.logsumexp(matrix, axis=-1, keepdims=True)
softmax = np.exp(matrix - logsumexp)

I understand that when you log equations that use division you would then subtract, i.e. log(1/2) = log(1) - log(2). However, in the implantation of the code above, shouldn't they also log the matrix in order to subract the logsumexp?

Answer 1

The idea of working in log-space to avoid underflow requires that the intermediate objects you use to track progress are themselves on the log-scale --- you only convert back to regular scale at the end of the computation. (If you use intermediate objects that are not on the log-scale then you introduce underflow at that point in the computation, which then defeats the whole purpose.) Consequently, I strongly suspect that the object matrix is using values that are already measured on the log-scale, and therefore you do not need to take a logarithm of the elements of this object in the code. I recommend you review the initial settings for this object to see if this is how things have been done.

Answer 2

matrix is on the log scale. If matrix were not on the log scale, then you would only want to do log(sum(matrix)) not logsumexp(matrix).

To see why, write down the steps and work through the algebra. It should be quite apparent if you do the manipulations.