I have modeled a distribution, $f$, over a r.v. $x \in \mathbb{R}^3$. At inference a set of measuring points, $X$, of the r.v. variables show up. I want to form a distribution over this sample set so that I can sample from it so that more probable points will show up more often.

My idea was to evaluate $f$ for each point in the set and then normalize it to form a discrete distribution over the points. Since $f \approx 0$ I want to use the log likelihood. However, the $\log$ is a nonlinear transform and normalizing it means calculating:

$\log \sum_i f(x_i)$,

which is bound to give numerical errors due to the fact that $f(x_i)\approx 0$. So the best I can do is a log sum exp calculation.

My question is: is there a better way of doing this? And is this a completely wrong approach in terms of modeling probability distributions?

  • 1
    $\begingroup$ You can't use log likelihood as a scaled density. (Hint: draw the log-likelihood for a single observation from a standard normal) $\endgroup$
    – Glen_b
    Sep 20, 2013 at 9:50
  • 2
    $\begingroup$ Maybe I missed something, but didn't you mean $log \prod_i f(x_i)$ rather than $log \sum_i f(x_i)$ ? $\endgroup$
    – Mr Renard
    Sep 20, 2013 at 9:57
  • $\begingroup$ If you have an un-normalized distribution that you want to normalize you take each element and divide by the sum. If your distribution is extremely spread out you have to do it in log space otherwise there will be numerical errors and therefore the log sum. $log\frac{p_i}{\sum_i p_i}$ $\endgroup$ Sep 20, 2013 at 11:30
  • $\begingroup$ The closely related thread at stats.stackexchange.com/questions/63447/… may answer parts (or all) of this question. $\endgroup$
    – whuber
    Sep 20, 2013 at 16:57

1 Answer 1


I'm not sure I understand things correctly. Here are my thoughts:

  1. If you have formed a likelihood for your data based on an assumption that each observation $x_i$ was a draw from a probability distribution which you have specified, then to generate similar samples you can just draw more samples from that distribution. So for example if you think each $x_i \sim MVN_3(0,I)$ then you can just draw more samples from a multi-variate Normal distribution (using R or some other Statistical software).
  2. If, on the other hand, you have a set of samples where each $x_i \sim f$, and you don't know what $f$ is, then it might be a good idea to look into Bootstrap re-sampling (see here: http://www.burns-stat.com/documents/tutorials/the-statistical-bootstrap-and-other-resampling-methods-2/ for a tutorial and here: http://en.wikipedia.org/wiki/Bootstrap_(statistics) for the wiki). Essentially you are drawing samples with replacement from your data, with a suitable degree of smoothing if necessary, which sounds a little like what you are describing. You can use the "sample" function in R (for example) to do this quite easily.

Does that help?

  • $\begingroup$ Yeah 2) does! It is difficult to phrase questions in a simple way without writing an essay. $\endgroup$ Sep 20, 2013 at 9:41
  • $\begingroup$ Yes I can imagine, especially when you don't know the exact thing you're looking for. Glad it was useful... $\endgroup$ Sep 20, 2013 at 9:49
  • $\begingroup$ Yeah 2) does! It is difficult to phrase questions in a simple way without writing an essay I realized. The problem with 1) is that it is in $R^3$ which takes forever to cover by sampling from $f$. My idea was that using the set of points as an estimate of more probable points in $R^3$ and using $f$ to approximate a discrete distribution over the set I can speed up the MAP inference I am trying to do. But bootstrapping seems promising. I had forgotten about that. Thanks! $\endgroup$ Sep 20, 2013 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.