1
$\begingroup$

I am looking at the formula of entropy on Wikipedia, where $P(X)$ is a probability mass function.

\begin{equation} H(X) = -\sum_{i=1}^{n}P(x_i)log_bP(x_i) \end{equation}

I got curious why they use capital $P$ here because I am used to seeing lower case $p$ for the probability of a specific event. So why is $P$ used instead of $p$? On the page of probability mass function they use other notations too

\begin{equation} p_X(x_i)=P(X=x_i) \end{equation}

but both of those notations are different from $P(x_i)$. Perhaps this is nit-picking but I feel that there is not a lot of leeway when using formulas correctly.

So is the notation in the entropy formula incorrect, or is it another valid alternative? If so, is $p(x_i)$ also correct?

$\endgroup$
1
  • $\begingroup$ A convention that I ocassionally see (and use myself) is to use upper case P(x) for probabilities and lower case p(x) for densities. $\endgroup$
    – cdalitz
    Jan 25, 2021 at 15:25

1 Answer 1

0
$\begingroup$

There are various notations out there. Some of them are commonly accepted and used, and some of them are rare but consistent within the context. The one in wiki page of PMF is pretty common and unambiguous.

In the entropy page, the formula uses lowercase $x_i$ (not uppercase). And, in the beginning of the article it says

Given a discrete random variable X, with possible outcomes $x_{1},...,x_{n}$, which occur with probability ${\displaystyle \mathrm {P} (x_{1}),...,\mathrm {P} (x_{n})}$, the entropy of X is formally defined as:

so, within the context, it is consistent. Instead of writing $P(X=x_i)$, it abuses notation and uses $P(x_i)$, but first defines it as such.

$\endgroup$
2
  • $\begingroup$ You are right, changed the $X$ to $x$. Could you say something about $p(x_i)$? Is that also valid in this context, or does it lack information? I assume it is not as precise as $P(X=x_i)$ because it is not obvious that $x_i$ is an event from the set of $X$? $\endgroup$ Jan 25, 2021 at 11:35
  • $\begingroup$ If not defined in the context, $p(x_i)$ is ambiguous. For example, the wikipedia article uses $p(x_i,y_j)$ but also defines it just afterwards (see conditional entropy formula). In "efficiency" section, it uses $p(x_i)$ but doesn't define it. it maybe the case that the authors of these sections were different. $\endgroup$
    – gunes
    Jan 25, 2021 at 11:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.