Timeline for How can I sample from a log transformed distribution using uniform distribution?
Current License: CC BY-SA 3.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 28, 2011 at 13:10 | comment | added | Karl | Great; I'm glad it worked, and thanks for the references. | |
| Oct 28, 2011 at 11:12 | comment | added | mahonya | Karl, you've actually given me the correct answer, but it took me some time to actually understand what you are suggesting. For anyone interested in a slightly more detailed explanation: facwiki.cs.byu.edu/nlp/index.php/Log_Domain_Computations and web.me.com/khallbobo/WebData/ethzfs2011nlp/logsum_details.pdf should help. Thanks! Ps: the only thing worth mentioning is that with the transformed probabilities, using log(U) would allow us to stay in the log domain all the time. Maybe I can build a sampler which returns a point from the original scale using all values from log scale. | |
| Oct 26, 2011 at 15:42 | comment | added | Karl | You ultimately need to get back to the real CDF; but once you've rescaled so that the density integrates to 1, there shouldn't be an underflow problem anymore, unless something really wacky is going on where the support of the density is enormous, but then I would re-scale X so that it doesn't occur. | |
| Oct 26, 2011 at 11:24 | vote | accept | mahonya | ||
| Oct 26, 2011 at 11:24 | comment | added | mahonya | Yes, but then does the inverse transform mechanism hold? It depends on the assumption that the CDF is between [0,1]. Even if I use addlog type of approach, the mapping from the range of Uniform[0,1] to CDF of the (log)density function is no longer valid I think I'll have to give up on this, but I'll accept your answer, since you've provided pretty much all the main options. Thanks! | |
| Oct 25, 2011 at 23:24 | comment | added | Karl |
You could look at modifying a function for numerical integration to work with the log density in place of the density, using something like the addlog function I mentioned in place of the usual addition.
|
|
| Oct 25, 2011 at 15:44 | comment | added | mahonya | log transforming densities, then normalizing them in the log scale (using log-sum-exp for denominator), then getting them back is all fine. However, finding the CDF requires that I integrate the function, which requires that I evaluate it (for numerical integration). Underflow is the reason I can't integrate it, hence, I need to find a way of performing CDF calculation in the log scale. In general, underflowing stops me from performing anything in the normal scale. | |
| Oct 24, 2011 at 18:25 | comment | added | Karl | But if you're going to make a draw from that distribution, don't you need to first re-scale it so that it integrates to 1 (or, once discretized, sums to 1)? That converts it from a joint distribution to a conditional distribution, at which point you shouldn't have the underflow problem. | |
| Oct 24, 2011 at 16:33 | comment | added | mahonya | I've tried to keep the question simple, so I might have avoided some necessary detail. I am using this method during Gibbs sampling to implement Griddy Gibbs ( jstor.org/stable/2290225 ) Each step in the gibbs sampling creates a univariate distribution, but the size of the observations causes the densities of points in this distribution to be extremely small. So my X only has a single dimension, but the density has very small values due to joint probability of observations, which are already known. I guess this particular sampling technique is problematic in this context. | |
| Oct 24, 2011 at 16:12 | comment | added | Karl | I don't think there's any way around calculating the cumulative distribution function; my first suggestion is a way to calculate the $\log c_i$ from the $\log p_i$, avoiding underflow. But if the problem raises from the high-dimensional nature of $X$, you might instead work conditional distributions rather than the joint distributions. Say $X = (X_1, \dots, X_n)$, then draw $X_1$ and then $X_2$ given $X_1$, and then ... $X_k$ given $(X_1, \dots, X_{k-1})$. | |
| Oct 24, 2011 at 16:02 | comment | added | mahonya | the reason my pi gets small is the joint likelihood of a high number of observations, so I have no way to avoid it. you are right about my method, but I can't say I've understood your suggestion. Is what you're suggesting an alternative to CDF (in the log space)? how does it replace Uniform distribution based method? Sorry, it may be obvious, but I'm not having the smartest days recently :) Thanks for the response! | |
| Oct 24, 2011 at 15:46 | history | answered | Karl | CC BY-SA 3.0 |