Why is it useful to sample probability distributions? I was reading the book Deep Learning by Goodfellow, Bengio & Courville and it seemed to imply that samples are only useful to:


*

*approximate sum/integrals (why is this sooo important?) 

*when the goal is to generate samples itself (which is trivial since of course its important to sample if that is the goal)


However, I remain unable to appreciate why sample is so important and why so much hard work has gone to study such a topic. Why is sampling important? Are there no other motivations? Are these really important enough on its own?
I'd love to be able to appreciate why sampling is an important topic.

My own thoughts
As someone inclined for ML, minimizing the expected loss is my goal:
$$ E_{x,y \sim p*_{x,y}} [Loss(f(X),Y)]$$
where $p^*$ is the true unknown distribution.
so I guess since this expectation is a sum or an integral we could try approximating the true generalization of our model if we could create more samples or create a model of the true distribution. This seems important, though it seems that this is not the approach people do for ML for some reason...

To provide further context on the exact extract I was reading from the deep learning book on the chapter on sampling (and Monte Carlo Methods) here is exact paragraph I was reading, title: 
Why Sampling:

There are many reasons that we may wish to draw samples from a
  probability distribution. Sampling provides a ﬂexible way to
  approximate many sums and integrals at reduced cost. Sometimes we use
  this to provide a signiﬁcant speedup toa costly but tractable sum, as
  in the case when we subsample the full training costwith minibatches.
  In other cases, our learning algorithm requires us to approximatean
  intractable sum or integral, such as the gradient of the log partition
  function ofan undirected model. In many other cases, sampling is
  actually our goal, in thesense that we want to train a model that can
  sample from the training distribution. (Chapter 17)

for me just reading that section equates to "drawing samples (from a model) is only useful to approximate sums/integrals and when you want to do sampling". For someone with much less of a statistics background, this justification seems quite shallow. I have seen a lot of mathematics and textbooks (like Koller's PGM book) devoted to sampling from models. This seems quite an important topic and it just seems that this book lacked a proper motivation for the why. This is where my question stems from.
 A: Often enough, we are not only interested in evaluating an integral, e.g., for calculating the expectation of a random variable, but in understanding the entire distribution, say for deriving quantiles. Which is important, e.g., in inventory control. And often enough, the underlying distribution is not analytically tractable, so sampling is the easiest and fastest way of going about this.
A simple example from my daily life: suppose we want to have a quantile forecast for retail sales. We believe than each day's sales are negative binomially distributed with known (forecasted) parameters. However, we don't need quantiles per day, but across, say, three or five days (because the truck arrives to fill up the store shelves twice a week, so each delivery has to cover multiple days). The sum of negbins is not analytically tractable, but it's trivial to simulate from each day's negbin, add the simulated values and get an appropriate quantile from the simulated sum that will achieve our desired service level.
(Plus, there are lots of other applications in cryptography etc. if you are really interested in why people invest so much effort in random-generation.)
A: In Bayesian statistics the denominator of the posterior, the "evidence", is an integral that's usually solved by estimation. Bayesian statistics is used to tune the hyper-parameters of neural networks.
In reinforcement learning, the expected total reward over time is an integral that's usually solved by estimation.
Also this: "ELFI is a statistical software package written in Python for performing inference with generative models. The term "likelihood-free inference" refers to a family of inference methods that replace the use of the likelihood function with a data generating simulator function. This is useful when the likelihood function is not computable or otherwise available but it is possible to make simulations of the process."
https://github.com/elfi-dev/elfi
A: Sampling is important, because if you can sample, you can evaluate expectations. But that's just to restate the motivation for the first question. So, to provide some examples (supplementing Stephen's answer) of what this implies:
Probabilities
Say you had no way to evaluate a distribution $F$ but could draw samples from it, then you can still answer questions about probabilities under $F$. When $X \sim F$ and $X^{(i)}$ are samples drawn from $F$ and $A$ is some sensible set of interest, $I$ the indicator function.
$\mathbb P(X \in A) = \mathbb E[I(X \in A)] \approx \frac{1}{N} \sum_{i=1}^N I(X^{(i)} \in A)$
So one reason sampling is so important is because it allows us to trade off difficult analytical problems (LHS) for a more tractable computational problems (RHS).
Make $A$ clicked ads, or the set of cat pics and you can get all kinds of useful things doing this.
Gradients
As another example, elsewhere in ML-land [1], sampling is used for gradient descent in the (rough) sense of, when $L$ is a loss function:
$\nabla L = \mathbb E[\nabla l(X)] \approx \frac{1}{N}  \sum_{i=1}^N \nabla l(X^{(i)})$
For some appropriate $l$ (sparing details, see the link). Again this lets us swap analysis for computation, which can be useful when the analysis is intractable.
[1] https://arxiv.org/abs/1506.03431
A: This is an important question to ask. I think you've hit a bit of a wall with it because of the common misconception that to draw a line between "sampling from a probability distribution" and "sampling from a population" is to create a false dichotomy. Let's look at a simple Bayesian example:
You flip a coin $n$ times and record $X$ heads. The number of heads in $n$ flips is given by the binomial distribution:
$$P(x|\theta)= \binom nx \theta^x (1-\theta)^{n-x}$$
We're interested in finding out plausible values for $\theta$, the probability of the coin landing Heads. We are interested in the posterior distribution of $\theta$:
$$p(\theta|x)=\frac{p(x|\theta)*p(\theta)}{A}$$
$A$ is just a normalizing constant, and yet it is often too complicated to solve for. This means we want to be able to sample from unnormalized distributions (distributions that do not add up to 1). So, instead of sampling from $p(\theta|x)$ we are interested in sampling from the unnormalized distribution $p(x|\theta)*p(\theta)$. Finally, we want to sample under different priors (different assumptions about the prior distribution $p(\theta)$).
If we assume 3 different priors for $\theta$, we want to be able to sample from  all 3 unnormalized probability distributions:
$$p(x|\theta)*p(\theta)=p(x|\theta)*prior_1$$
$$p(x|\theta)*p(\theta)=p(x|\theta)*prior_2$$
$$p(x|\theta)*p(\theta)=p(x|\theta)*prior_3$$
What this example tried to illustrate is that we need the ability to sample from any nonnegative function that we may want to write down on paper; I don't call it a probability distribution because some may think this refers to sampling from named distributions only (Normal, Binomial, Poisson, etc.); yet in practice we are sampling from some unnamed complicated nonnegative function. The other reason I do not call it a probability distribution is so that we do not mistake our mission for that of sampling from normalized distributions only; very often we need to sample from unnormalized distributions.
Now, finally, I am ready to actually answer your question. The reason it is important to learn how to sample from any nonnegative function we may write down (unnormalized probability distribution) is that real life phenomena are very complex. We model that complexity with a statistical model. Whether with a Monte Carlo simulation or a Bayesian statistical model (e.g. Hierarchical Model), at the end of the day we are interested in some random variable that more often than not has a very complicated probability distribution. To learn about our random variable of interest, we need to be able to sample from its complicated (likely unnormalized) probability distribution. And that is why sampling methods such as Metropolis-Hastings are so important to all of science. Important enough that it was named one of the 10 most important algorithms of the 20th century.
