I have a random process that follows a normal distribution. It's parameters are mean = 35 units, std.dev. = 8 units. I've seen from the wiki entry for the normal distribution that there is a formula to calculate the entropy. So plugging in the figures as:- $$ .5\log\left(2\pi e^1 8\cdot 8\right) $$ I get a value of 1.52, which I take to be per sample. My question is what are these units? What thing do I have 1.52 of?

Information entropy is (typically) measured in units of bits, after Claude Shannon's definition. So can I take it that each sample generates 1.52 bits of entropy? Clearly recording those samples generates information and therefore occupies a real and discrete amount of storage space. Ergo entropy cannot be unit less.

  • 4
    $\begingroup$ entropy is unitless $\endgroup$ – Aksakal Sep 27 '17 at 2:12
  • $\begingroup$ Yes, unless you meant something else by "unit" $\endgroup$ – Aksakal Sep 27 '17 at 3:26
  • 1
    $\begingroup$ However the $\sigma$ may be in meters. Do we have something like $\log $ square meters? $\endgroup$ – Karel Macek Sep 27 '17 at 4:01
  • $\begingroup$ You link to information entropy, which is shannon entropy, that is, the discrete case. You ask about entropy for the normal distribution, that is differential entropy, something entirely different. $\endgroup$ – kjetil b halvorsen Sep 27 '17 at 9:45
  • $\begingroup$ The point is that shannon and differential entropy has very different properties and must be treated separately $\endgroup$ – kjetil b halvorsen Sep 27 '17 at 9:56

@Aksakal gave the answer in comments, entropy is unitless. Still, there are some details to elaborate. First, sometimes units are given as "nats" or "bits", for the cases of use of natural logs/binary logs, respectively. But these are not really measurements units (like meter, kg, ...) in the physical sense, it corresponds more to writing the measurement in decimal or binary number systems. A measurement of length in meters can be written in decimal or binary, that does not really change the unit of measurement of length used.

Some details: We treat Shannon (discrete) and differential (continuous) entropy separately. $$ \DeclareMathOperator{\E}{\mathbb{E}} H(X) = -\sum_x p(x) \log p(x) = -\E_X \log p(X) $$ where $p$ is the probability mass function of a discrete random variable. Then $$ H_d(X) = -\int f(x) \log f(x) \; dx = -\E_X \log f(X) $$ where $f$ is the probability density function of a continuous random variable. Now, from general principles the unit of measurement of the expectation (mean, average) of a variable (random or not) is the same as the unit of measurement of the variable itself. This leaves us with the unit of measurement of $\log p(x), \log f(x)$ respectively. Again, from general principles (see lognormal distribution, standard-deviation and (physical) units for discussion and references) the arguments of transcendental functions like $\log$ must be unitless. That rises a problem, while $p(x)$ certainly is unitless, since probability is an absolute number, the density $f(x)$ measures probability pr unit of $x$, so if unit of $x$ is $\text{u}$, then unit of $f(x)$ is $\text{u}^{-1}$. So, for the equation defining differential entropy $H_d$ to be dimensionally correct, we must assume the argument to log contains a "hidden" constant with numerical value 1 and unit $\text{u}$. But the conclusion follows, that both Shannon and differential entropy is unitless. Still, one must remember that differential entropy scales with the unit of measurement of $X$, as discussed in https://en.wikipedia.org/wiki/Differential_entropy

| cite | improve this answer | |
  • $\begingroup$ But Shannon entropy is measured in bits, and absolutely not unit less. I'm not sure where you get your idea of what it is, but Shannon (who invented information entropy) says it's measured in bits as he wrote in his A Mathematical Theory of Communication paper. $\endgroup$ – Paul Uszak Oct 9 '17 at 20:40
  • $\begingroup$ Did you read my answer? Bits and nats is mentioned, as "units", but they are really only a description of the base of logs used. You can compare it with "radians", used as name of a unit , angle measure, but as angle measure is a quotient of two length, is really unitless. Bits, nats, like radians, is not a true unit like meter is $\endgroup$ – kjetil b halvorsen Oct 9 '17 at 21:08
  • $\begingroup$ Here mathforum.org/library/drmath/view/64034.html is a link to a good discussion of the "parallell" issue of the unit of measurement of angles, as radians. $\endgroup$ – kjetil b halvorsen Oct 12 '17 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.