# Why are log probabilities useful?

Probabilities of a random variable's observations are in the range $$[0,1]$$, whereas log probabilities transform them to the log scale. What then is the corresponding range of log probabilities, i.e. what does a probability of 0 become, and is it the minimum of the range, and what does a probability of 1 become, and is this the maximum of the log probability range? What is the intuition of this being of any practical use compared to $$[0,1]$$?

I know that log probabilities allow for stable numerical computations such as summation. However, besides arithmetic, how does this transformation improve applications compared to the case where raw probabilities are used instead? A comparative example for a continuous random variable before and after logging would be good

• Log of probability zero is just log of zero as usual, and so indeterminate. That's not fatal to other uses. With entropy we say $p \log p \equiv 0$ whenever $p = 0$, which can be justified more rigorously. Logarithms can be useful to the extent that probabilities multiply, and for other reasons. The logarithm of a probability density can be useful too. So the logarithm of a Gaussian density is a quadratic and other distributions may helpfully be compared on log scale, as that stretches smaller densities. Log of cumulative probability or survival probability can be useful too. Aug 20, 2020 at 14:34
• New questions deserve ... new questions. Aug 20, 2020 at 14:38
• No; as said, comments are not for new questions. It's easy enough to find discussions of that. Also, while entropy (Shannon sense) is based on probabilities and log probabilities it is neither a probability nor a log probability. Aug 20, 2020 at 14:45
• related: log-odds en.wikipedia.org/wiki/Logit
– qwr
Aug 21, 2020 at 3:53
• @NickCox "Indeterminate" isn't quite the right word for $\log (0)$. In this context, we can identify the symbol $\log(0)$ with the limit $\lim_{x\to0^+} \log(0) = -\infty$ in a determinate way, meaning that $\log(0) = \log(0)$ "makes sense" in a way that "indeterminate" expressions like 0/0 don't. For example, we can't say that $1/2 = \lim_{x\to0}\frac{x}{2x} = 0/0 = \lim_{x\to0}\frac{x}{x} = 1$ because $0/0$ is indeterminate. The expression $\log(0)$ does not have this ambiguity with the use of the = sign, making it determinate, despite not being a real number. Aug 21, 2020 at 22:17

The log of $$1$$ is just $$0$$ and the limit as $$x$$ approaches $$0$$ (from the positive side) of $$\log x$$ is $$-\infty$$. So the range of values for log probabilities is $$(-\infty, 0]$$.

The real advantage is in the arithmetic. Log probabilities are not as easy to understand as probabilities (for most people), but every time you multiply together two probabilities (other than $$1 \times 1 = 1$$), you will end up with a value closer to $$0$$. Dealing with numbers very close to $$0$$ can become unstable with finite precision approximations, so working with logs makes things much more stable and in some cases quicker and easier. Why do you need any more justification than that?

• is there more behind the intuition explaining how a measure that is negative and unbounded has come to be more preferable to one that is already bounded between $[0,1]$? Aug 20, 2020 at 14:34
• Log scale is not preferable and I don't think @Greg Snow is saying that either. It is just useful as explained. Aug 20, 2020 at 14:37
• @develarist as the answer already mentions, if you want to represent a very, very, very small probability, then in the commonly used digital representations (e.g. IEEE 754 floating point numbers) it's preferable to store them as huge negative numbers in the logarithmic representation instead of very small positive numbers close to 0 in the direct representation, since in the latter case you'll have larger numerical errors in every calculation caused by the difference between the true value and the closest value that can be represented with the finite precision used in that encoding. Aug 20, 2020 at 23:21
• @develarist Since log(x*y)=log(x)+log(y), doing calculations of joint probablities in 'log-space' is trivial and numerically accurate. Aug 21, 2020 at 15:55
• @develarist of course, some representation error is unavoidable, however, there's a quantitative difference in how much these errors accumulate over all the calculations you are going to make, since you get some extra representation error in each intermediate value, and so the scale of these (many) intermediate values and the resulting errors matter more than the scale and representation error of the initial input. Often there's a limited need to convert between log-probabilities and actual probabilities, e.g. if I need argmax or beam search, it can be done directly it in log-space. Aug 24, 2020 at 20:25

Taking the log of a probability or probability density can often simplify certain computations, such as calculating the density gradient given some of its parameters. This is particularly true when the density belongs to the exponential family: often, taking the log allows for straightfowad. This makes taking the derivative by hand simpler (as product rules become simpler sum rules) and can also lead to more stable numerical derivative calculations, such as finite differencing.

As an illustration, let's take the Poisson with probability function $$f_x=e^{-\lambda}\frac{\lambda^{x}}{x!}.$$ While $$x$$ is a discrete variable, the function is smooth with respect to $$\lambda$$. Applying a logarithmic transformation, the function becomes $$\log f_x= -\lambda + x\log(\lambda) - \log(x!),$$ and its derivative with respect to $$\lambda$$ is $$\frac{\partial \log f_x}{\partial \lambda} = -1 + \frac{x}{\lambda},$$ an expression involving just two simple operations. Now, contrast this result with the partial derivative of $$f_x$$: $$\frac{\partial f_x}{\partial \lambda} = \frac{e^{-\lambda } (x-\lambda) \lambda ^{x-1}}{x!}:$$ the calculation involves natural exponentiation, real exponentiation, computation of a factorial, and, worst of all, division by a factorial. This involves more computation time and less numerical stability, even in this simple example. The result is compounded for more complex probability functions, as well as when observing an i.i.d. sample of random variables since these are added in log space while multiplied in probability space (again, complicating derivative calculation, as well as introducing more of the floating point error mentioned in the other answer).

These gradient expressions play a crucial role in the analytic and numerical determination of Maximum a Posteriori ($$\ell_0$$ Bayes) and Maximum Likelihood Estimators. They also facilitate the numerical resolution of the Method of Moments estimating equations, frequently through Newton's method, which requires computations of Hessians or second derivatives. The complexity difference between logged and unlogged forms can be substantial in this context. Moreover, these expressions aid in demonstrating the equivalence between least squares and maximum likelihood when dealing with a Gaussian error structure.

• the log(x!) term should be subtracting and not adding when taking the logarithm of the Poisson probability function Aug 27, 2020 at 1:32
• @RubenGarcia thank you. Aug 27, 2020 at 15:25

As an example of the process mentioned in Greg Snow's answer: I quite often use high-level programming languages (Octave, Maxima[*], Gnuplot, Perl,...) to compute ratios between marginal likelihoods for Bayesian model comparison. If one tries to compute the ratio of marginal likelihoods directly, intermediate steps in the calculation (and sometimes the final result too) very frequently go beyond the capabilities of the floating-point number implementation in the interpreter/compiler, producing numbers so small that the computer can't tell them apart from zero, when all the important information is in the fact that those numbers are actually not quite zero. If, on the other hand, one works in log probabilities throughout, and takes the difference between the logarithms of the marginal likelihoods at the end, this problem is much less likely to occur.

[*] Sometimes, Maxima evades the problem by using rational-number arithmetic instead of floating-point arithmetic, but one can't necessarily rely on this.

This might not be what you are interested in, but log probabilities in statistical physics are closely related to the concepts of energy and entropy. For a physical system in equilibrium at temperature $$T$$ (in kelvin), the difference in energy between two microstates A and B is related to the logarithm of the probabilities that the system is in state A or state B:

$$E_\mathrm{A} - E_\mathrm{B} =-k_\mathrm{B}T \left[ \ln(P_\mathrm{A}) - \ln( P_\mathrm{B}) \right]$$

So, statistical physicists often work with log probabilities (or scaled versions of them), because they are physically meaningful. For example, the potential energy of a gas molecule in an atmosphere at a fixed temperature under a uniform gravitation field (a good approximation near the surface of the Earth) is $$mgh$$, where $$m$$ is the mass of the gas molecule, $$g$$ is the acceleration of gravity, and $$h$$ is the height of the molecule above the surface. The probability of finding a gas molecule in the top floor of the building versus in the bottom floor (assuming the floors have the same volume and the floor-to-ceiling height is small) is given by:

$$mg (h_\mathrm{top} - h_\mathrm{bottom}) \approx -k_\mathrm{B} T \left[ \ln (P_\mathrm{top}) - \ln(P_\mathrm{bottom}) \right]$$

This probability is trivially related to the concentration of the gas on the two floors. Higher floors have a lower concentration and the concentration of heavier molecules decays more quickly with height.

In statistical physics, it is often useful to switch back and forth between quantities proportional to log probabilities (energy, entropy, enthalpy, free energy) and quantities proportional to probability (number of microstates, partition function, density of states).