In node-based measures of semantic similarity, information content ($IC$) of a given term ($c$) can be computed as: $$IC(c) = -log~p(c)$$

Where $p(c)$ is the probability of occurrence of term $c$ in a corpus.

This method, described by Resnik, has been applied to biomedical ontologies. In this context, investigators can calculate $p(c)$ as the the probability of occurrence of a term $c$ being annotated. Although $c$ may exist in the corpus, it's possible that $c$ has $0$ annotations (ex. a gene with unknown function -- gene is in corpus, but unannotated).

Assuming $p(c)$ is the probability a term is annotated in a corpus:

$p(c_1) = 0.50$

$p(c_2) = 0.10$

$p(c_3) = 0.00$

Yields the following $IC$:

$ICc_1 = 0.693$

$ICc_2 = 2.303$

$ICc_3 = undefined$

The specific problem of $ICc_3 = undefined$ has been approached by maximizing $IC$ for unannotated terms and documented here. Excerpt:

From: http://bmcgenomics.biomedcentral.com/articles/10.1186/1471-2164-15-1091

Alternatively, I've seen authors set $IC=0$ for unannotated terms, one example, Excerpt:

enter image description here

Can anyone explain the motivation of the authors setting $IC = 0$ for terms that are unannotated in the second link? I'm struggling with this sharp contrast against the intuition that infrequently annotated terms should have higher $IC$. Or perhaps just shed some light on the fundamental differences and how to best avoid $-log(0)$?


Batet, et al. propose an alternative approach to calculating information content where "1 is added to the numerator and denominator to avoid log(0)".
Link to .pdf


2 Answers 2


The simplest most common way to avoid a 0 probability in word frequencies is the Lidstone smoothing

Which is basically, instead of using $$p(w_i)=\frac{\#(w_i)}{\sum{\#(w_j)}}$$ Use: $$p(w_i)=\frac{\#(w_i)+\epsilon}{\sum{\#(w_j)}+N\epsilon}$$

Regarding information of $p=0-$ The motivation I know is taken from the entropy definition: $$H(p)=p\log{p}$$ And when $$p\rightarrow0$$ We can prove that $$H(p)\rightarrow0$$ Using l'hopital rule

  • $\begingroup$ Thanks for the link & summary. Smoothing seems to be generally applicable, and more similar to the first authors' method. $\endgroup$
    – Kevin
    Commented Oct 25, 2016 at 15:04

I think the appropriate answer will depend on what you want to use the "information content" for in the end.

I personally do not like the justification given for the second case, i.e. "R gives log(0)=NaN". Mathematically we would have $$\mathrm{IC}[0]=-\log[0]=\infty$$ so to me the more relevant consideration would be the reason that the term has $p=0$.

Why would this make sense? If a word is very common in the corpus ($p\approx{1}$) then intuitively it is not very informative ($\mathrm{IC}\approx{0}$), i.e. knowing that a term occurs in a document tells us very little about that document specifically. Conversely, if a term is very rare in the corpus ($p\ll{1}$) we might expect the term to be very informative ($\mathrm{IC}\gg{0}$), i.e. documents in the corpus that contain that term are "distinctive" somehow. Following this logic, then in the limit of $p=0$ we might say that if we see the missing term in a new document, we might say that document is "infinitely distinct" from the corpus.

Why might this be a problem? I can think of two cases:

  • If a term is "expected" in the corpus (i.e. you are bothering to record a $p$ for it), then it could be missing due to the finite sample size. In this case your "$p=0$" is more of a "below measurement threshold", so really you have censored data, i.e. $p<\frac{1}{N}$ for an $N$ document corpus. In this case your information content is also censored, i.e. $\mathrm{IC}>\log{N}$. The first approach you mention seems to be based on similar reasoning (i.e. $\mathrm{IC}[0]\equiv\max_{p>0}\mathrm{IC}[p]$).

or, alternatively:

  • The term is so common that it would have $p=1$, but this was known beforehand, so it was simply excluded from the analysis (i.e. a stop word). This would be a rational basis for the second approach you mention (i.e. $\mathrm{IC}[0]\equiv{0}$)
  • $\begingroup$ Thanks, I agree, the second case mentioned above confuses me a bit. In the context of biomedical ontologies, the censored data approach makes more sense than the stop word rationale. $\endgroup$
    – Kevin
    Commented Oct 25, 2016 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.