9
$\begingroup$

I've seen this answer: How is it logically possible to sample a single value from a continuous distribution?, but it's still not very clear to me.

In Scipy, there's a function scipy.stats.norm.rvs() which samples from a normal distribution.

I was trying to understand how it works, imagining that "under the hood" we're really sampling from a discrete Random Variable at some arbitrary level of granularity.

However, our professor explained that we can always sample values from a continuous distribution, and that "The density represents the range of different values with the associated likelihood of them occurring".

I'm having trouble reconciling that statement with the fact that if $X$ is a continuous RV, then $P(X=x) = 0$.

If $P(X=x) = 0$, how can the density value of the PDF give a likelihood of a value, or range of values, occurring? What am I missing?

Improve this question
$\endgroup$
5
  • $\begingroup$ I don't understand your professor's statement. Actually I might but it's pretty confusingly worded. And I don't understand what it is exactly that you're confused about. Maybe it's the fact that densities aren't probabilities. You have to integrate densities to get probabilities. If you integrate over any region, there will never be any contradiction. In particular you will never get mass on a point. $\endgroup$
    – Taylor
    Commented Sep 19, 2016 at 0:49
  • 1
    $\begingroup$ @Taylor I don't understand the statement either, unfortunately...what I'm confused about is how we can sample single values form a continuous distribution. In the Scipy function I mentioned, the function is sampling an array of single values from a normal distribution. I'm just wondering how that's possible. Is it possible in theory and I'm misunderstanding something? Or is it that the normal distribution is actually a histogram (discrete RV) "under the hood" that we're sampling from when using that function? I understand what you're saying about integrating, but single values aren't regions. $\endgroup$
    – jeremy radcliff
    Commented Sep 19, 2016 at 0:53
  • $\begingroup$ It's discrete under the hood, but not for reasons that you've mentioned. Google pseudorandom number generators. And well, they are, but they wil always have 0 probability. Probability is before you see the data. Likelihood is after $\endgroup$
    – Taylor
    Commented Sep 19, 2016 at 0:54
  • 1
    $\begingroup$ @Taylor, I understand what you're saying about integration of course, so are you confirming that the idea of sampling single values from a continuous distribution is not something that's defined properly? Or just impossible? What I mean is that no matter how small your area of integration, it's not a single value; it's still an interval of infinitely many values. $\endgroup$
    – jeremy radcliff
    Commented Sep 19, 2016 at 0:56
  • $\begingroup$ These are two separate issues: how to code something, and what it means to just passively observe random data. If we're talking about the second one, I don't see a contradiction. Probabilities don't describe your data, since it isn't random anymore $\endgroup$
    – Taylor
    Commented Sep 19, 2016 at 0:59

2 Answers 2

Reset to default
9
$\begingroup$

In practice the functions that sample from continuous distributions at best sample only to some level of accuracy. For example, if we're sampling from a uniform on the unit interval, typically what happens is there's an algorithm that samples uniformly over some (very large) range of integers (say $0,1,...,m-1$) and these may be converted to numbers in $[0,1)$ by dividing by $m$. So you can see $n/m$ or $(n-1)/m$ or $(n+1)/m$ but not values in between.

If you think of these discrete values as representing values within a range ($n/m$ in some sense "stands for" values in $[n/m, (n+1)/m)$ then there's a sense in which the sampled values could be regarded as standing for an interval of truly continuous values; while such considerations become somewhat more complex once you start transforming them, nevertheless in many situations the endpoints of the intervals can be tracked through such transformations and the process maintained as needed.

Note that your professor's comment doesn't seem to be talking about what is usually done but rather what we could do. In that case whuber's comment at the linked post is relevant:

One (inefficient) way is to generate each successive binary digit independently until the number is known sufficiently precisely for the calculations.

One way to look at that is that we can (as before) regard any current representation as a proxy for an interval of values but we can generate as many additional digits as required when we need them. In that case, a given generated value has always only been partly generated; the process of generating to more precision may be undertaken as the precision is needed.

In reality all our representations of continuous quantities (not just in random generation, but in any measurement) are limited in accuracy; normally this doesn't do any harm to our notions of continuous variables as a suitable model for what we are doing.

Improve this answer
$\endgroup$
6
  • $\begingroup$ If I understand what you're saying in the last 2 paragraphs, let's say you broke down your unit interval into 100 (for easy ex) segments and now want a degree of granularity to the 1000th, you could just use whatever value you've already sampled and append to it a randomly sampled single decimal place? $\endgroup$
    – jeremy radcliff
    Commented Sep 19, 2016 at 2:17
  • 4
    $\begingroup$ Yes, if we were working in base 10 (on a computer you could do it in binary rather than decimal naturally). You can imagine doing it by rolling a 10-sided die numbered $0-9$ (such dice exist); then say you roll a "$2$" which represents $[0.2,0.3)$. Then you discover that for some purpose you need to know if the value is more than $2/9$; you can't yet tell. So you generate an additional digit, "$7$", so your number is $[0.27,0.28)$. Clearly that's greater than $2/9$, so you could stop there. But if you'd rolled a "2" you would still be in doubt and would need to roll a third time... ctd $\endgroup$
    – Glen_b
    Commented Sep 19, 2016 at 2:25
  • 2
    $\begingroup$ ctd... You don't have to restrict yourself to a single digit, of course - you could generate more if you wanted - but it's possible to get there only generating a digit at a time in whatever base is convenient. $\endgroup$
    – Glen_b
    Commented Sep 19, 2016 at 2:25
  • $\begingroup$ Thank you for confirming with an example. Just one last thing; now the way I think about this is that sampling is an "in practice" concept, not a mathematical one, so mathematically it's meaningless to sample from a continuous RV since $P(X=x)=0$. However in practice, we're not claiming really to sample from a continuous RV; what we're doing is saying that we can approximate the population we're interested in by, for ex, a normal distribution and then sampling from what is really a discrete RV (based on the continuous RV) whose degree of granularity we can extend as needed. Is that reasonable? $\endgroup$
    – jeremy radcliff
    Commented Sep 19, 2016 at 2:39
  • 2
    $\begingroup$ If you always extend that granularity as far as needed (even if only as far as needed), how would you distinguish if from actual continuity? It's only the fact that we would deliberately stop short of taking it as far as needed that would make it actually granular. $\endgroup$
    – Glen_b
    Commented Sep 19, 2016 at 3:33
5
$\begingroup$

You are correct that random number generators are really sampling from a discrete, granular distribution. Floating point numbers have only finite precision (8 bit, 16 bit, etc.) and computers can only generate floating point numbers within a specified precision. Generating a truly random complete real variable would require specifying an infinite amount of information, and isn't necessary -- finite precision results are, with care, more than precise enough.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.