5
$\begingroup$

I know this is an extremely basic question, but I have never had a course on statistics or applied probability. The only probability I had was in a measure theory course. Now I am doing machine learning and I am seeing the phrase "sample data from distribution" everywhere, and I have only a very vague idea as to what it means.

  • Part of the problem is that sampling sorts of defies everyday intuition. Suppose we want to "sample from a uniform distribution" which means to me "choose things at random". Can this really be done?

    Suppose you place several balls in a line and tell a person to pick one "at random", I think intuitively, most people will either pick the first, or the last one or in the middle but not somewhere in between. In other words, our idea of random as experienced in everyday life is not "mathematically random", because there are behavioral patterns we can observe. This is also why if you tell conduct experiment telling a group of people to pick a number between 1 to 10, the most common answer is not 3, or 8, but something like 5.

    So I don't understand sampling because it is not my everyday experience. I have never truly seen someone sample anything with a uniform distribution, let alone a Gumbel, Gaussian or Geometric or some other distribution.

  • Reality aside, how would this be done numerically. Suppose I present you the statement:

    "We sample data $x$ from a Gaussian distribution"

    What does this mean exactly? A distribution is the probability density of a random variable, a mathematical function it tells me the probability of finding a data point $x$ over an area. All it provides me is a very vague idea as to how the data may look like, i.e., where I am most likely to find a data point. A probability density function doesn't tell me at all how many data points are there, where they are placed exactly, etc. Now what does it mean to sample a data, mathematically speaking? What is actually done when "picking a data point from or according to a distribution"?

  • Finally, data points generated from experiments are always finite, and always discrete. There is probably nothing in nature that is continuous or with uncountable cardinality. Sure, we can model the distribution of these data points via a continuous probability density function. But isn't the the probability of finding any point is zero according to a Gaussian i.e., $\Pr[X = x] = 0, X \sim \mathcal{N}(\mu,\sigma^2)$?. So wouldn't it be impossible to "get" any point according to a probability distribution, since you never know exactly where they are?

Can someone show me how you would actually "sample from a distribution"?

As a hypothetical scenario, I have $20$ data points, $x_i \in \mathbb{R}^2$, I put them somewhere on the Euclidean plane. Suppose you have no idea about their location, all you know is a probability density model of these data points generated from some algorithm. Actually, let's make it harder: you also have no idea about the dimensionality or the number of data points. Now I say, sample a data point $x_i$ from these $N$ number of data points via the "insert name here" distribution and return that point $x_i$ to me, and prove it to me that you have sampled this data point from the distribution you claim it was sampled from. How would you actually do it?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Have you ever watched a Bingo game, a state-sponsored lottery drawing, or a casino game that draws cards, rolls dice, or spins a wheel? All those (when done fairly) select uniformly at random from a space of possible outcomes. Those outcomes are all labeled with numbers, some of them repeated (as with the cards): those labelings can produce non-uniform distributions of the numbers that are drawn. Those are time-honored, practical ways to sample from distributions. $\endgroup$ – whuber♦ Dec 9 '17 at 18:42
  • 1
    $\begingroup$ Simply speaking, "drawing a sample $x$ from a distribution $p$" means to "somehow produce" an $x$ with probability equal to $p(x)$. You can imagine having an algorithm that produces some $x$, and when you run it infinitely many times, you see a particular $x$ with frequency $p(x)$ as dictated by the distribution. The issue of how to construct such an algorithm given arbitrary $p$ is a problem studied in statistics (easy examples of $p$ include uniform or discrete distributions). $\endgroup$ – Yibo Yang Nov 1 '18 at 19:11
3
$\begingroup$

1) Whether humans can sample without bias is a question entirely different from whether random sampling can be done. Yes, random sampling can be done, and though some will argue that random seeds are partially deterministic, for all intents and purposes a computer generated random sample is random enough.

2) What does 'randomly sample from x distribution' mean? In short, it means collecting a set of N points that were generated by some theoretical distribution. For the normal distribution, assuming a given mu and sigma, you select N points that conform to those parameters. I realize this is an unsatisfactory answer, but consider that many algorithms begin by sampling from the uniform distribution, which is fairly straightforward to comprehend. U(0,5) will produce a number between 0 and 5, each with equal probability. With a few steps, you can draw these random numbers and ensure they conform to the gaussian as detailed in: sampling from normal

It is very common to sample from particular distributions, for demonstration and simulation purposes. Most popular statistics packages have these functions built-in: in R, you have rnorm, rbinom, rpois, runif, etc. If you were to sample a sizeable dataset using these functions, then try to fit it with any theoretical distribution, you'd find that the best fitting would match the one that generated it.

3) I do not think one can say that a datapoint from an experiment is always discrete. A discrete variable can only take on particular values, and a mean of a particular measure might take any number of significant digits. But, you are right that strictly speaking, the probability of any one particular, exact value in a probability density is vanishingly small; which is why one often uses the CDF rather than the PDF (the probability that a value is less than or equal to some value is easier to calculate).

4) Your last question and challenge is ill-formed. One cannot hope to prove or even guess the generating distribution from a single point. You'd need some non-trivial sample size to do that (in which case, you could very well do this).

|cite|improve this answer
$\endgroup$
  • $\begingroup$ So in order to sample from a distribution, you must have a set of data that already conforms to a certain distribution. Suppose you are presented with $N$ number of points, you calculate the mean and the covariance matrix of these points, and from these two parameters, you can model using a Gaussian PDF. Then would it be correct to then say, drawing any point from these set of points is equivalent to "sampling according to the Gaussian distribution"? Seems to be too convenient, however, since Gaussian is the only PDF that can be completely determined through the mean and the covariance.matrix. $\endgroup$ – Shamisen Expert Dec 9 '17 at 7:37
  • $\begingroup$ No, you do not need to already have a distribution in order to sample from that distribution. The PDF generating function provides the rules for 'creating' a unique dataset that conforms to it. Also, even if you HAD a little 'gaussian' dataset from which to draw, your new subsample would actually have smaller variance than you'd hope (this is actually the reason for the n-1 sample variance correction). $\endgroup$ – HEITZ Dec 9 '17 at 7:44
  • $\begingroup$ The link to "sampling from normal" is broken. $\endgroup$ – nbro Jul 24 '18 at 12:51
  • $\begingroup$ @nbro - Here is a link that works for me: stats.stackexchange.com/questions/16334/… $\endgroup$ – n1k31t4 May 9 '19 at 15:17
2
$\begingroup$

This could really be answered by taking a course in Monte Carlo simulation, which goes into this topic in depth. Here's a good set of slides that cover common approaches to generating samples from specific distributions. There are also more complicated approaches like Markov Chain Monte Carlo and copula methods for very complicated distributions with dependence and other behavior.

This is a well studied area that should not be hard to remedy, especially if you've been able to handle measure theory. This is simple stuff in comparison.

Now, mathematically, why do these methods work? At their heart they depend on the ability to generate a sequence of real-valued numbers $X_i$ in the interval $[0,1]$ such that

$$\lim_{n \to \infty} \frac{|\{i: X_i \in (a,b), i\leq n\}|}{n} = b-a$$

In addition, we demand that $|COV(X_i,X_j)| \approx 0, \forall i\neq j$

There are a ton more mathematically stringent tests that such sequences must pass to demonstrate statistical randomness(see here, and here).

"Random numbers" are actually *pseudo-*random numbers -- they are deterministically created but are statistically indistinguishable from iid observations from a uniform distribution (up to some enormous lag).

Using the pseudo-random numbers, we can generate sequences of numbers in a range like $[0,1]$ and then use various transforms and algorithms to turn these into all the other types of random variables and processes we use.

Why $[0,1]$? Well, as someone who knows measure theory, you know that $P(\Omega)=1, P(\emptyset)=0, 1\geq P(X|X\subset \Omega) \geq 0$, so drawing random samples on this range allows you to sample the probability measure's range.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for the first link, I am more comfortable with the idea of drawing from an uniform distribution. But can you say something general from the data drawn from one distribution versus another? Suppose I draw a data point from exponential distribution, and another from Gaussian distribution (You don't have to use these examples). Is there any way to "smell" how the data was drawn? That is a basic question isn't it, otherwise I can present you with any data point and claim that it was drawn from a distribution and you'd have no way to verify if this is true or not. $\endgroup$ – Shamisen Expert Dec 9 '17 at 7:28
  • 1
    $\begingroup$ @StackexchangeHouseNinja no...data points can only speak for their values, not how they got there. I'd only know if you showed me your sampling procedure or data generating algorithm. Now, as you collect more data, certain distributions become increasingly implausible, so there is a sense in which we can asymptotically tell which distribution a dataset came from by via the convergence of its ECDF to the CDF. $\endgroup$ – user145807 Dec 9 '17 at 7:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.