What does it mean to sample a data point from or according to a distribution? [duplicate]

Question

I know this is an extremely basic question, but I have never had a course on statistics or applied probability. The only probability I had was in a measure theory course. Now I am doing machine learning and I am seeing the phrase "sample data from distribution" everywhere, and I have only a very vague idea as to what it means.

• Part of the problem is that sampling sorts of defies everyday intuition. Suppose we want to "sample from a uniform distribution" which means to me "choose things at random". Can this really be done?

  Suppose you place several balls in a line and tell a person to pick one "at random", I think intuitively, most people will either pick the first, or the last one or in the middle but not somewhere in between. In other words, our idea of random as experienced in everyday life is not "mathematically random", because there are behavioral patterns we can observe. This is also why if you tell conduct experiment telling a group of people to pick a number between 1 to 10, the most common answer is not 3, or 8, but something like 5.

  So I don't understand sampling because it is not my everyday experience. I have never truly seen someone sample anything with a uniform distribution, let alone a Gumbel, Gaussian or Geometric or some other distribution.

• Reality aside, how would this be done numerically. Suppose I present you the statement:

  "We sample data $x$ from a Gaussian distribution"

  What does this mean exactly? A distribution is the probability density of a random variable, a mathematical function it tells me the probability of finding a data point $x$ over an area. All it provides me is a very vague idea as to how the data may look like, i.e., where I am most likely to find a data point. A probability density function doesn't tell me at all how many data points are there, where they are placed exactly, etc. Now what does it mean to sample a data, mathematically speaking? What is actually done when "picking a data point from or according to a distribution"?

• Finally, data points generated from experiments are always finite, and always discrete. There is probably nothing in nature that is continuous or with uncountable cardinality. Sure, we can model the distribution of these data points via a continuous probability density function. But isn't the the probability of finding any point is zero according to a Gaussian i.e., $\Pr[X = x] = 0, X \sim \mathcal{N}(\mu,\sigma^2)$?. So wouldn't it be impossible to "get" any point according to a probability distribution, since you never know exactly where they are?

Can someone show me how you would actually "sample from a distribution"?

As a hypothetical scenario, I have $20$ data points, $x_i \in \mathbb{R}^2$, I put them somewhere on the Euclidean plane. Suppose you have no idea about their location, all you know is a probability density model of these data points generated from some algorithm. Actually, let's make it harder: you also have no idea about the dimensionality or the number of data points. Now I say, sample a data point $x_i$ from these $N$ number of data points via the "insert name here" distribution and return that point $x_i$ to me, and prove it to me that you have sampled this data point from the distribution you claim it was sampled from. How would you actually do it?

Answer 1

1) Whether humans can sample without bias is a question entirely different from whether random sampling can be done. Yes, random sampling can be done, and though some will argue that random seeds are partially deterministic, for all intents and purposes a computer generated random sample is random enough.

2) What does 'randomly sample from x distribution' mean? In short, it means collecting a set of N points that were generated by some theoretical distribution. For the normal distribution, assuming a given mu and sigma, you select N points that conform to those parameters. I realize this is an unsatisfactory answer, but consider that many algorithms begin by sampling from the uniform distribution, which is fairly straightforward to comprehend. U(0,5) will produce a number between 0 and 5, each with equal probability. With a few steps, you can draw these random numbers and ensure they conform to the gaussian as detailed in: sampling from normal

It is very common to sample from particular distributions, for demonstration and simulation purposes. Most popular statistics packages have these functions built-in: in R, you have rnorm, rbinom, rpois, runif, etc. If you were to sample a sizeable dataset using these functions, then try to fit it with any theoretical distribution, you'd find that the best fitting would match the one that generated it.

3) I do not think one can say that a datapoint from an experiment is always discrete. A discrete variable can only take on particular values, and a mean of a particular measure might take any number of significant digits. But, you are right that strictly speaking, the probability of any one particular, exact value in a probability density is vanishingly small; which is why one often uses the CDF rather than the PDF (the probability that a value is less than or equal to some value is easier to calculate).

4) Your last question and challenge is ill-formed. One cannot hope to prove or even guess the generating distribution from a single point. You'd need some non-trivial sample size to do that (in which case, you could very well do this).

Answer 2

This could really be answered by taking a course in Monte Carlo simulation, which goes into this topic in depth. Here's a good set of slides that cover common approaches to generating samples from specific distributions. There are also more complicated approaches like Markov Chain Monte Carlo and copula methods for very complicated distributions with dependence and other behavior.

This is a well studied area that should not be hard to remedy, especially if you've been able to handle measure theory. This is simple stuff in comparison.

Now, mathematically, why do these methods work? At their heart they depend on the ability to generate a sequence of real-valued numbers $X_i$ in the interval $[0,1]$ such that

$$\lim_{n \to \infty} \frac{|\{i: X_i \in (a,b), i\leq n\}|}{n} = b-a$$

In addition, we demand that $|COV(X_i,X_j)| \approx 0, \forall i\neq j$

There are a ton more mathematically stringent tests that such sequences must pass to demonstrate statistical randomness(see here, and here).

"Random numbers" are actually *pseudo-*random numbers -- they are deterministically created but are statistically indistinguishable from iid observations from a uniform distribution (up to some enormous lag).

Using the pseudo-random numbers, we can generate sequences of numbers in a range like $[0,1]$ and then use various transforms and algorithms to turn these into all the other types of random variables and processes we use.

Why $[0,1]$? Well, as someone who knows measure theory, you know that $P(\Omega)=1, P(\emptyset)=0, 1\geq P(X|X\subset \Omega) \geq 0$, so drawing random samples on this range allows you to sample the probability measure's range.