6
$\begingroup$

I am transforming an unscaled density function to log scale to avoid underflow issues.

BI was performing integration on this function on a grid of values before I used the log transormation, to build a grid based cumulative distribution function. Then, using a draw from uniform[0,1], I was choosing the the largest point on the grid which had a cumulative probability value smaller than the draw from the uniform. This worked fine as long as the univariate density function could be integrated.

With the transform to log-scale, I can't really get my head around this mechanism. The joint likelihood is so small that I can't back transform the density, so I have to perform the same uniform distribution based sampling on the log scale. Is this an established practice? Feedback and pointers would be appreciated.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

As I understand it, you've generally discretized to create a set of $n$ points, $x_1, \dots, x_n$, with probability $p_1, \dots, p_n$, and you then calculate the cumulative probabilities, say $c_i = \sum_{j=1}^i p_j$. So you can draw $U \sim Uniform(0,1)$ and then take $X = x_{i^*}$ where $i^* = \min_i \{i:c_i \ge U\}$, or something like that.

But your current problem is that the $p_i$ are so small that you want to just work with $a_i = \log p_i$.

One approach would be to sort the $a_i$ from largest to smallest and then calculate partial sums using something like the following addlog function, which calculates $\log(f + g)$ on the basis of $a = \log(f)$ and $b = \log(g)$. 

addlog(a, b, THRESH=200.0)
{
  if(b > a + THRESH) return(b);
  else if(a > b + THRESH) return(a);
  else return(a + log1p(exp(b-a)));
}

where log1p(x) returns log(1+x).

But, really, I would think that you should focus on the $x_i$ for which $p_i$ is large enough that you don't need to worry about underflow, and neglect the $x_i$ with exceedingly small $p_i$. If all of the $p_i$ are small, then it seems that you should grid more coarsely. In most applications, it should be sufficient to discretize to 1000 or so values, I would think.

Improve this answer
$\endgroup$
10
  • $\begingroup$ the reason my pi gets small is the joint likelihood of a high number of observations, so I have no way to avoid it. you are right about my method, but I can't say I've understood your suggestion. Is what you're suggesting an alternative to CDF (in the log space)? how does it replace Uniform distribution based method? Sorry, it may be obvious, but I'm not having the smartest days recently :) Thanks for the response! $\endgroup$
    – mahonya
    Commented Oct 24, 2011 at 16:02
  • $\begingroup$ I don't think there's any way around calculating the cumulative distribution function; my first suggestion is a way to calculate the $\log c_i$ from the $\log p_i$, avoiding underflow. But if the problem raises from the high-dimensional nature of $X$, you might instead work conditional distributions rather than the joint distributions. Say $X = (X_1, \dots, X_n)$, then draw $X_1$ and then $X_2$ given $X_1$, and then ... $X_k$ given $(X_1, \dots, X_{k-1})$. $\endgroup$
    – Karl
    Commented Oct 24, 2011 at 16:12
  • $\begingroup$ I've tried to keep the question simple, so I might have avoided some necessary detail. I am using this method during Gibbs sampling to implement Griddy Gibbs ( jstor.org/stable/2290225 ) Each step in the gibbs sampling creates a univariate distribution, but the size of the observations causes the densities of points in this distribution to be extremely small. So my X only has a single dimension, but the density has very small values due to joint probability of observations, which are already known. I guess this particular sampling technique is problematic in this context. $\endgroup$
    – mahonya
    Commented Oct 24, 2011 at 16:33
  • $\begingroup$ But if you're going to make a draw from that distribution, don't you need to first re-scale it so that it integrates to 1 (or, once discretized, sums to 1)? That converts it from a joint distribution to a conditional distribution, at which point you shouldn't have the underflow problem. $\endgroup$
    – Karl
    Commented Oct 24, 2011 at 18:25
  • $\begingroup$ log transforming densities, then normalizing them in the log scale (using log-sum-exp for denominator), then getting them back is all fine. However, finding the CDF requires that I integrate the function, which requires that I evaluate it (for numerical integration). Underflow is the reason I can't integrate it, hence, I need to find a way of performing CDF calculation in the log scale. In general, underflowing stops me from performing anything in the normal scale. $\endgroup$
    – mahonya
    Commented Oct 25, 2011 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.