2
$\begingroup$

I am trying to implement the following:

enter image description here

where the right part returns a probability between 0 and 1. Regarding the product, the authors of the respective paper note:

Due to numerical precision issues with products of probabilities, in our implementation we follow common practice and use summation of log probabilities.

Form what I understand, using the sum of log probabilities helps to prevent underflow. But then I do not get a value between 0 and 1 and the 1- in the formula above does not make sense. What am I missing here? And can I transform the sum of log probabilities back to a value between 0 and 1? When using a large number of probabilities, I still get a very small number, e.g.:

log_probability = math.log(0.9) + math.log(0.3) + math.log(0.9) + math.log(1) + math.log(0.9) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) + math.log(0.3) 
prob = math.exp(log_probability)

Where log_probality=-15.967728003210647 and prob =1.1622614669999998e-07.

Thank you, I am really hitting the edge of my understanding of stats here...!

$\endgroup$
7
  • $\begingroup$ Implementation questions might be more on topic on Stack Overflow. $\endgroup$ May 2 '20 at 16:03
  • $\begingroup$ It looks like you're using Python... Option (1) List comprehensions. Option (2) Built-in map function. Option (3) NumPy-vectorized functions. Option 3 is what I would recommend looking at first.. $\endgroup$ May 2 '20 at 16:04
  • $\begingroup$ For NumPy arrays you can set the datatype, including the float size that is related to the precision. $\endgroup$ May 2 '20 at 16:06
  • $\begingroup$ Yes, I was thinking that this is rather about my (lack of) understanding of the stats which is why I decided to put it here. $\endgroup$
    – Steinroe
    May 2 '20 at 16:06
  • $\begingroup$ If you're really in need of more precision than what NumPy can offer then you might consider mpmath for arbitrary float precision (AFP). AFP is expensive, but sometimes needed. $\endgroup$ May 2 '20 at 16:08
2
$\begingroup$

Working in log-probabilties

Yeah, the probability is still small if you apply the inverse transform (exponentiate the sum of log-probability). One can work in log probabilities to avoid really large/small probabilities that can result numerical issues, including underflow. You may or may not transform back to probabilities, as the result of a sequence of operations may not yield a log-probability whose corresponding probability is within your float precision.

A lot of statistics makes use of mathematical optimization, and in many cases if you optimize an expression in terms of log-probability, you are also finding the same (or corresponding) optima for an original problem posed in terms of probability. For example, if we wanted to find the parameter $\theta$ that maximizes $P(Y | X, \theta)$ where $Y$ and $X$ are random variables, we might consider working with $\log \left[ P(Y | X, \theta) \right]$ to find the same optimal value of $\theta$, which we might denote as $\theta^*$.

$$\theta^* = \arg \max_\theta P(Y | X, \theta) = \arg \max_\theta \log \left[ P(Y | X, \theta) \right]$$

$\endgroup$
4
  • $\begingroup$ Thanks for your replies! Maybe to provide more context: Its about anomaly detection and the formula above describes the anomaly likelihood. So from what I understand, the above formula with 1-(a product of probabilities) can be "replaced" with the sum of the log probabilities as it behaves similar and is easier to work with in terms of computation? How does the 1- come into play here? With the above formula I threshold the resulting value and if exceeds a certain threshold, an anomaly is given. $\endgroup$
    – Steinroe
    May 3 '20 at 11:18
  • $\begingroup$ If we're looking for optimal parameters, we can drop constant terms when we wish as they preserve argmax/argmin. For example, $f(x) = x^2 + 1$ and $g(x) = x^2$ are both minimized where $x=0$ even though $f(0) = 1$ and $g(0) = 0$ are not equal to each other. $\endgroup$ May 3 '20 at 15:05
  • $\begingroup$ I skimmed the paper, and it looks like $L_t$ is a likelihood function, so it would make sense to apply this type of thinking about optimization. $\endgroup$ May 3 '20 at 15:09
  • $\begingroup$ If your just finding these likelihoods through some estimators of the mean, stdev (which I saw in the paper), then you can still drop the "1" provided that you adjust how you're thresholding accordingly. $\endgroup$ May 3 '20 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.