# Summation of Log Probabilities

I am trying to implement the following:

where the right part returns a probability between 0 and 1. Regarding the product, the authors of the respective paper note:

Due to numerical precision issues with products of probabilities, in our implementation we follow common practice and use summation of log probabilities.

Form what I understand, using the sum of log probabilities helps to prevent underflow. But then I do not get a value between 0 and 1 and the 1- in the formula above does not make sense. What am I missing here? And can I transform the sum of log probabilities back to a value between 0 and 1? When using a large number of probabilities, I still get a very small number, e.g.:

log_probability = math.log(0.9) + math.log(0.3) + math.log(0.9) + math.log(1) + math.log(0.9) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3)
prob = math.exp(log_probability)


Where log_probality=-15.967728003210647 and prob =1.1622614669999998e-07.

Thank you, I am really hitting the edge of my understanding of stats here...!

• Implementation questions might be more on topic on Stack Overflow. May 2 '20 at 16:03
• It looks like you're using Python... Option (1) List comprehensions. Option (2) Built-in map function. Option (3) NumPy-vectorized functions. Option 3 is what I would recommend looking at first.. May 2 '20 at 16:04
• For NumPy arrays you can set the datatype, including the float size that is related to the precision. May 2 '20 at 16:06
• Yes, I was thinking that this is rather about my (lack of) understanding of the stats which is why I decided to put it here. May 2 '20 at 16:06
• If you're really in need of more precision than what NumPy can offer then you might consider mpmath for arbitrary float precision (AFP). AFP is expensive, but sometimes needed. May 2 '20 at 16:08

A lot of statistics makes use of mathematical optimization, and in many cases if you optimize an expression in terms of log-probability, you are also finding the same (or corresponding) optima for an original problem posed in terms of probability. For example, if we wanted to find the parameter $$\theta$$ that maximizes $$P(Y | X, \theta)$$ where $$Y$$ and $$X$$ are random variables, we might consider working with $$\log \left[ P(Y | X, \theta) \right]$$ to find the same optimal value of $$\theta$$, which we might denote as $$\theta^*$$.
$$\theta^* = \arg \max_\theta P(Y | X, \theta) = \arg \max_\theta \log \left[ P(Y | X, \theta) \right]$$
• If we're looking for optimal parameters, we can drop constant terms when we wish as they preserve argmax/argmin. For example, $f(x) = x^2 + 1$ and $g(x) = x^2$ are both minimized where $x=0$ even though $f(0) = 1$ and $g(0) = 0$ are not equal to each other. May 3 '20 at 15:05
• I skimmed the paper, and it looks like $L_t$ is a likelihood function, so it would make sense to apply this type of thinking about optimization. May 3 '20 at 15:09