# Why is it useful to sample probability distributions?

I was reading the book Deep Learning by Goodfellow, Bengio & Courville and it seemed to imply that samples are only useful to:

1. approximate sum/integrals (why is this sooo important?)
2. when the goal is to generate samples itself (which is trivial since of course its important to sample if that is the goal)

However, I remain unable to appreciate why sample is so important and why so much hard work has gone to study such a topic. Why is sampling important? Are there no other motivations? Are these really important enough on its own?

I'd love to be able to appreciate why sampling is an important topic.

My own thoughts

As someone inclined for ML, minimizing the expected loss is my goal:

$$E_{x,y \sim p*_{x,y}} [Loss(f(X),Y)]$$

where $p^*$ is the true unknown distribution.

so I guess since this expectation is a sum or an integral we could try approximating the true generalization of our model if we could create more samples or create a model of the true distribution. This seems important, though it seems that this is not the approach people do for ML for some reason...

To provide further context on the exact extract I was reading from the deep learning book on the chapter on sampling (and Monte Carlo Methods) here is exact paragraph I was reading, title:

Why Sampling:

There are many reasons that we may wish to draw samples from a probability distribution. Sampling provides a ﬂexible way to approximate many sums and integrals at reduced cost. Sometimes we use this to provide a signiﬁcant speedup toa costly but tractable sum, as in the case when we subsample the full training costwith minibatches. In other cases, our learning algorithm requires us to approximatean intractable sum or integral, such as the gradient of the log partition function ofan undirected model. In many other cases, sampling is actually our goal, in thesense that we want to train a model that can sample from the training distribution. (Chapter 17)

for me just reading that section equates to "drawing samples (from a model) is only useful to approximate sums/integrals and when you want to do sampling". For someone with much less of a statistics background, this justification seems quite shallow. I have seen a lot of mathematics and textbooks (like Koller's PGM book) devoted to sampling from models. This seems quite an important topic and it just seems that this book lacked a proper motivation for the why. This is where my question stems from.

• Could you identify any scientific study anywhere that does not rely on a sample? That might help us understand what you might mean by "samples."
– whuber
Commented May 23, 2017 at 22:29
• @whuber maybe I should re-phrase, why is sampling from a model important? Of course we need sample given by the true process (frequentist mindset) to do anything. Maybe what I mean is, what is the point of sampling if we can just rely on the true true distribution for "correct" samples. Why would we need to make up samples? Commented May 23, 2017 at 22:45
• I am struggling to understand what you are talking about. Could you indicate what the context, situation, or application is that you have in mind?
– whuber
Commented May 23, 2017 at 23:27
• We have a site with (quite literally) thousands of questions which involve sampling distributions in one way or another (some search and research may be needed for your question). Can you focus your question a bit? Commented May 24, 2017 at 1:01
• @whuber I provided the exact paragraph from the book I am reading, hope it helps provide further context. What I am really going for is more for the why of sampling (from a model $p_{model}(x)$) rather than the how. There seems to be plenty of literature on the how, but very few comments on the why. For example, if I made a probabilistic model to sample more images seems like a silly model if we can get natural images really easy. This is just a silly example, though. Commented May 24, 2017 at 3:02

Often enough, we are not only interested in evaluating an integral, e.g., for calculating the expectation of a random variable, but in understanding the entire distribution, say for deriving quantiles. Which is important, e.g., in inventory control. And often enough, the underlying distribution is not analytically tractable, so sampling is the easiest and fastest way of going about this.

A simple example from my daily life: suppose we want to have a quantile forecast for retail sales. We believe than each day's sales are negative binomially distributed with known (forecasted) parameters. However, we don't need quantiles per day, but across, say, three or five days (because the truck arrives to fill up the store shelves twice a week, so each delivery has to cover multiple days). The sum of negbins is not analytically tractable, but it's trivial to simulate from each day's negbin, add the simulated values and get an appropriate quantile from the simulated sum that will achieve our desired service level.

(Plus, there are lots of other applications in cryptography etc. if you are really interested in why people invest so much effort in .)

• sorry if this is a trivial question but why are quantiles super important? That random-generation like you gave sent me to a lot of questions...is there one I should read or do you mind providing a comment on crypto? Commented May 23, 2017 at 21:47
• Using Stephan's example, let's say you want to have a 75% chance (or pick any other number you like) not to run out of stock before the following shipment arrives. How much will you need to order for the next few days? Commented May 24, 2017 at 0:59
• @Glen_b already gave a good explanation for why quantiles are important in inventory control. Simply put, you don't only want enough milk in the supermarket to cover average demand - you want (say) 95% of shoppers to get what they want, not run out of product after half the shoppers have arrived. You need quantiles for that. Regarding crypto, I'm not a expert - Crypto.SE might be useful. Commented May 24, 2017 at 6:20

This is an important question to ask. I think you've hit a bit of a wall with it because of the common misconception that to draw a line between "sampling from a probability distribution" and "sampling from a population" is to create a false dichotomy. Let's look at a simple Bayesian example:

You flip a coin $$n$$ times and record $$X$$ heads. The number of heads in $$n$$ flips is given by the binomial distribution:

$$P(x|\theta)= \binom nx \theta^x (1-\theta)^{n-x}$$

We're interested in finding out plausible values for $$\theta$$, the probability of the coin landing Heads. We are interested in the posterior distribution of $$\theta$$: $$p(\theta|x)=\frac{p(x|\theta)*p(\theta)}{A}$$ $$A$$ is just a normalizing constant, and yet it is often too complicated to solve for. This means we want to be able to sample from unnormalized distributions (distributions that do not add up to 1). So, instead of sampling from $$p(\theta|x)$$ we are interested in sampling from the unnormalized distribution $$p(x|\theta)*p(\theta)$$. Finally, we want to sample under different priors (different assumptions about the prior distribution $$p(\theta)$$).

If we assume 3 different priors for $$\theta$$, we want to be able to sample from all 3 unnormalized probability distributions:

$$p(x|\theta)*p(\theta)=p(x|\theta)*prior_1$$ $$p(x|\theta)*p(\theta)=p(x|\theta)*prior_2$$ $$p(x|\theta)*p(\theta)=p(x|\theta)*prior_3$$

What this example tried to illustrate is that we need the ability to sample from any nonnegative function that we may want to write down on paper; I don't call it a probability distribution because some may think this refers to sampling from named distributions only (Normal, Binomial, Poisson, etc.); yet in practice we are sampling from some unnamed complicated nonnegative function. The other reason I do not call it a probability distribution is so that we do not mistake our mission for that of sampling from normalized distributions only; very often we need to sample from unnormalized distributions.

Now, finally, I am ready to actually answer your question. The reason it is important to learn how to sample from any nonnegative function we may write down (unnormalized probability distribution) is that real life phenomena are very complex. We model that complexity with a statistical model. Whether with a Monte Carlo simulation or a Bayesian statistical model (e.g. Hierarchical Model), at the end of the day we are interested in some random variable that more often than not has a very complicated probability distribution. To learn about our random variable of interest, we need to be able to sample from its complicated (likely unnormalized) probability distribution. And that is why sampling methods such as Metropolis-Hastings are so important to all of science. Important enough that it was named one of the 10 most important algorithms of the 20th century.

In Bayesian statistics the denominator of the posterior, the "evidence", is an integral that's usually solved by estimation. Bayesian statistics is used to tune the hyper-parameters of neural networks.

In reinforcement learning, the expected total reward over time is an integral that's usually solved by estimation.

Also this: "ELFI is a statistical software package written in Python for performing inference with generative models. The term "likelihood-free inference" refers to a family of inference methods that replace the use of the likelihood function with a data generating simulator function. This is useful when the likelihood function is not computable or otherwise available but it is possible to make simulations of the process."

https://github.com/elfi-dev/elfi

Sampling is important, because if you can sample, you can evaluate expectations. But that's just to restate the motivation for the first question. So, to provide some examples (supplementing Stephen's answer) of what this implies:

Probabilities

Say you had no way to evaluate a distribution $F$ but could draw samples from it, then you can still answer questions about probabilities under $F$. When $X \sim F$ and $X^{(i)}$ are samples drawn from $F$ and $A$ is some sensible set of interest, $I$ the indicator function.

$\mathbb P(X \in A) = \mathbb E[I(X \in A)] \approx \frac{1}{N} \sum_{i=1}^N I(X^{(i)} \in A)$

So one reason sampling is so important is because it allows us to trade off difficult analytical problems (LHS) for a more tractable computational problems (RHS).

Make $A$ clicked ads, or the set of cat pics and you can get all kinds of useful things doing this.

As another example, elsewhere in ML-land [1], sampling is used for gradient descent in the (rough) sense of, when $L$ is a loss function:
$\nabla L = \mathbb E[\nabla l(X)] \approx \frac{1}{N} \sum_{i=1}^N \nabla l(X^{(i)})$
For some appropriate $l$ (sparing details, see the link). Again this lets us swap analysis for computation, which can be useful when the analysis is intractable.