Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
From what I learned in introductory statistics, the probability of getting any single value for a variable described by a probability density distribution is 0, since the integral under a single value is 0. However, realistically, if I wanted to sample from a distribution I should get a single value back.
Can anyone explain this?
Imagine that you have a bowl and inside of this bowl there's an infinite number of colorful balls. Probability of randomly drawing some particular ball is so small, that it is practically zero. Yet, this doesn't mean that you cannot draw a ball from this bowl.
Another example, imagine that you are throwing darts into a dartboard. If you consider a chance of hitting a particular point on the dartboard (up to infinite precision), then the chance is again zero. Yet, if you threw a dart, you'll certainly hit some point (unless you miss the dartboard...).
There is an uncountably infinite number of values in the support of a continuous distribution. The computer however can only represent a final ammount of real nubers (it opreates with a floating-point representation of $\mathbb{R}$). In this representation tiny intervals of real numbers are mapped to a single number (say in the middle of the interval) hence you obtain not a draw from the distribution but its floating-point image. Ties when simulating large samples of random variables from some distributions are not uncommon:
> set.seed(21)
> sum(duplicated(runif(10^7)))
[1] 11569
> set.seed(21)
> sum(duplicated(runif(10^5)))
[1] 1
What the computer generates are pseudo-random numbers after all. It depends on the needs of your particular application whether the generated sample is satisfying.
In general, what you sample (in computer systems) is a number with finite precision. For example, if the number is represented by 32 bits, then in equally likely case, the probability of obtaining that number is $1/2^{32}$. If the precision increases, the probability of obtaining a specific value decreases. In infinite precision, it becomes $0$. The numbers generated by True random number generators via a measurement process are also observed in digital systems, i.e. when someone/something tells you the generated number, it uses a finite precision. Or, it might be the case that the measurement process might be finite precision.
