If I have a continuous probability distribution, say normal, it's clearly theoretically continuous and values can be anywhere within it's range (+/- infinity I guess to infinite precision).

I then take samples always to a precision of two decimal places, say 0.00 to 5.00. Is the distribution continuous or discrete then for the purposes of further calculations like entropy? Surely it must be discrete as there are only 500 values it can take. But then that would mean there is no such thing as a continuous distribution in the real world once it's sampled which feels wrong.

I suspect that this is schoolboy level confusion but it's been too long since I was one...

  • 1
    $\begingroup$ Usually for the purposes of modelling and calculating entropy, you assume that the data comes from some distribution. Just because you are measuring up to a precision of two decimal places doesn't mean the actual data can't have infinitely many numbers after the decimal. A continuous distribution definitely can exist in the real world. But I do agree that because we only take samples of distributions we can never be 100% sure of that. $\endgroup$ – user3494047 Mar 1 '17 at 2:49
  • $\begingroup$ @user3494047 That's exactly what I don't understand. Data is generated by the observer, not the source distribution. For entropy calculations, surely it only matters how many decimal places your samples have. More digits -> more entropy -> more storage space required. So for information entropy calculations, all distributions have to be discrete and relate to the measurement precision? $\endgroup$ – Paul Uszak Mar 1 '17 at 3:06
  • 1
    $\begingroup$ Calculate the entropy for the continuous distribution, then calculate it again from your probability table of 500 values. Is it close enough? $\endgroup$ – kjetil b halvorsen Mar 1 '17 at 7:18
  • $\begingroup$ @kjetilbhalvorsen Would you mind expanding on "probability table of 500 " ? Are you saying that I should treat it as a discrete probability distribution, rather than continuous? $\endgroup$ – Paul Uszak Mar 5 '17 at 0:54
  • $\begingroup$ What else can you do? That would be an approximation to the integral in the continuous case, yes? $\endgroup$ – kjetil b halvorsen Mar 5 '17 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.