4
$\begingroup$

I was reading the book Deep Learning by Goodfellow, Bengio & Courville and it seemed to imply that samples are only useful to:

  1. approximate sum/integrals (why is this sooo important?)
  2. when the goal is to generate samples itself (which is trivial since of course its important to sample if that is the goal)

However, I remain unable to appreciate why sample is so important and why so much hard work has gone to study such a topic. Why is sampling important? Are there no other motivations? Are these really important enough on its own?

I'd love to be able to appreciate why sampling is an important topic.


My own thoughts

As someone inclined for ML, minimizing the expected loss is my goal:

$$ E_{x,y \sim p*_{x,y}} [Loss(f(X),Y)]$$

where $p^*$ is the true unknown distribution.

so I guess since this expectation is a sum or an integral we could try approximating the true generalization of our model if we could create more samples or create a model of the true distribution. This seems important, though it seems that this is not the approach people do for ML for some reason...


To provide further context on the exact extract I was reading from the deep learning book on the chapter on sampling (and Monte Carlo Methods) here is exact paragraph I was reading, title:

Why Sampling:

There are many reasons that we may wish to draw samples from a probability distribution. Sampling provides a flexible way to approximate many sums and integrals at reduced cost. Sometimes we use this to provide a significant speedup toa costly but tractable sum, as in the case when we subsample the full training costwith minibatches. In other cases, our learning algorithm requires us to approximatean intractable sum or integral, such as the gradient of the log partition function ofan undirected model. In many other cases, sampling is actually our goal, in thesense that we want to train a model that can sample from the training distribution. (Chapter 17)

for me just reading that section equates to "drawing samples (from a model) is only useful to approximate sums/integrals and when you want to do sampling". For someone with much less of a statistics background, this justification seems quite shallow. I have seen a lot of mathematics and textbooks (like Koller's PGM book) devoted to sampling from models. This seems quite an important topic and it just seems that this book lacked a proper motivation for the why. This is where my question stems from.

$\endgroup$
  • 3
    $\begingroup$ Could you identify any scientific study anywhere that does not rely on a sample? That might help us understand what you might mean by "samples." $\endgroup$ – whuber May 23 '17 at 22:29
  • 1
    $\begingroup$ @whuber maybe I should re-phrase, why is sampling from a model important? Of course we need sample given by the true process (frequentist mindset) to do anything. Maybe what I mean is, what is the point of sampling if we can just rely on the true true distribution for "correct" samples. Why would we need to make up samples? $\endgroup$ – Pinocchio May 23 '17 at 22:45
  • 3
    $\begingroup$ I am struggling to understand what you are talking about. Could you indicate what the context, situation, or application is that you have in mind? $\endgroup$ – whuber May 23 '17 at 23:27
  • 2
    $\begingroup$ We have a site with (quite literally) thousands of questions which involve sampling distributions in one way or another (some search and research may be needed for your question). Can you focus your question a bit? $\endgroup$ – Glen_b May 24 '17 at 1:01
  • $\begingroup$ @whuber I provided the exact paragraph from the book I am reading, hope it helps provide further context. What I am really going for is more for the why of sampling (from a model $p_{model}(x) $) rather than the how. There seems to be plenty of literature on the how, but very few comments on the why. For example, if I made a probabilistic model to sample more images seems like a silly model if we can get natural images really easy. This is just a silly example, though. $\endgroup$ – Pinocchio May 24 '17 at 3:02
2
$\begingroup$

Often enough, we are not only interested in evaluating an integral, e.g., for calculating the expectation of a random variable, but in understanding the entire distribution, say for deriving quantiles. Which is important, e.g., in inventory control. And often enough, the underlying distribution is not analytically tractable, so sampling is the easiest and fastest way of going about this.

A simple example from my daily life: suppose we want to have a quantile forecast for retail sales. We believe than each day's sales are negative binomially distributed with known (forecasted) parameters. However, we don't need quantiles per day, but across, say, three or five days (because the truck arrives to fill up the store shelves twice a week, so each delivery has to cover multiple days). The sum of negbins is not analytically tractable, but it's trivial to simulate from each day's negbin, add the simulated values and get an appropriate quantile from the simulated sum that will achieve our desired service level.

(Plus, there are lots of other applications in cryptography etc. if you are really interested in why people invest so much effort in .)

$\endgroup$
  • $\begingroup$ sorry if this is a trivial question but why are quantiles super important? That random-generation like you gave sent me to a lot of questions...is there one I should read or do you mind providing a comment on crypto? $\endgroup$ – Pinocchio May 23 '17 at 21:47
  • $\begingroup$ Using Stephan's example, let's say you want to have a 75% chance (or pick any other number you like) not to run out of stock before the following shipment arrives. How much will you need to order for the next few days? $\endgroup$ – Glen_b May 24 '17 at 0:59
  • 2
    $\begingroup$ @Glen_b already gave a good explanation for why quantiles are important in inventory control. Simply put, you don't only want enough milk in the supermarket to cover average demand - you want (say) 95% of shoppers to get what they want, not run out of product after half the shoppers have arrived. You need quantiles for that. Regarding crypto, I'm not a expert - Crypto.SE might be useful. $\endgroup$ – Stephan Kolassa May 24 '17 at 6:20
1
$\begingroup$

In Bayesian statistics the denominator of the posterior, the "evidence", is an integral that's usually solved by estimation. Bayesian statistics is used to tune the hyper-parameters of neural networks.

In reinforcement learning, the expected total reward over time is an integral that's usually solved by estimation.

Also this: "ELFI is a statistical software package written in Python for performing inference with generative models. The term "likelihood-free inference" refers to a family of inference methods that replace the use of the likelihood function with a data generating simulator function. This is useful when the likelihood function is not computable or otherwise available but it is possible to make simulations of the process."

https://github.com/elfi-dev/elfi

$\endgroup$
1
$\begingroup$

Sampling is important, because if you can sample, you can evaluate expectations. But that's just to restate the motivation for the first question. So, to provide some examples (supplementing Stephen's answer) of what this implies:

Probabilities

Say you had no way to evaluate a distribution $F$ but could draw samples from it, then you can still answer questions about probabilities under $F$. When $X \sim F$ and $X^{(i)}$ are samples drawn from $F$ and $A$ is some sensible set of interest, $I$ the indicator function.

$\mathbb P(X \in A) = \mathbb E[I(X \in A)] \approx \frac{1}{N} \sum_{i=1}^N I(X^{(i)} \in A)$

So one reason sampling is so important is because it allows us to trade off difficult analytical problems (LHS) for a more tractable computational problems (RHS).

Make $A$ clicked ads, or the set of cat pics and you can get all kinds of useful things doing this.

Gradients

As another example, elsewhere in ML-land [1], sampling is used for gradient descent in the (rough) sense of, when $L$ is a loss function:

$\nabla L = \mathbb E[\nabla l(X)] \approx \frac{1}{N} \sum_{i=1}^N \nabla l(X^{(i)})$

For some appropriate $l$ (sparing details, see the link). Again this lets us swap analysis for computation, which can be useful when the analysis is intractable.

[1] https://arxiv.org/abs/1506.03431

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.