I know this is an extremely basic question, but I have never had a course on statistics or applied probability. The only probability I had was in a measure theory course. Now I am doing machine learning and I am seeing the phrase "sample data from distribution" everywhere, and I have only a very vague idea as to what it means.
Part of the problem is that sampling sorts of defies everyday intuition. Suppose we want to "sample from a uniform distribution" which means to me "choose things at random". Can this really be done?
Suppose you place several balls in a line and tell a person to pick one "at random", I think intuitively, most people will either pick the first, or the last one or in the middle but not somewhere in between. In other words, our idea of random as experienced in everyday life is not "mathematically random", because there are behavioral patterns we can observe. This is also why if you tell conduct experiment telling a group of people to pick a number between 1 to 10, the most common answer is not 3, or 8, but something like 5.
So I don't understand sampling because it is not my everyday experience. I have never truly seen someone sample anything with a uniform distribution, let alone a Gumbel, Gaussian or Geometric or some other distribution.
Reality aside, how would this be done numerically. Suppose I present you the statement:
"We sample data $x$ from a Gaussian distribution"
What does this mean exactly? A distribution is the probability density of a random variable, a mathematical function it tells me the probability of finding a data point $x$ over an area. All it provides me is a very vague idea as to how the data may look like, i.e., where I am most likely to find a data point. A probability density function doesn't tell me at all how many data points are there, where they are placed exactly, etc. Now what does it mean to sample a data, mathematically speaking? What is actually done when "picking a data point from or according to a distribution"?
Finally, data points generated from experiments are always finite, and always discrete. There is probably nothing in nature that is continuous or with uncountable cardinality. Sure, we can model the distribution of these data points via a continuous probability density function. But isn't the the probability of finding any point is zero according to a Gaussian i.e., $\Pr[X = x] = 0, X \sim \mathcal{N}(\mu,\sigma^2)$?. So wouldn't it be impossible to "get" any point according to a probability distribution, since you never know exactly where they are?
Can someone show me how you would actually "sample from a distribution"?
As a hypothetical scenario, I have $20$ data points, $x_i \in \mathbb{R}^2$, I put them somewhere on the Euclidean plane. Suppose you have no idea about their location, all you know is a probability density model of these data points generated from some algorithm. Actually, let's make it harder: you also have no idea about the dimensionality or the number of data points. Now I say, sample a data point $x_i$ from these $N$ number of data points via the "insert name here" distribution and return that point $x_i$ to me, and prove it to me that you have sampled this data point from the distribution you claim it was sampled from. How would you actually do it?