# Measuring Information Content of unannotated terms in a corpus, avoiding -log(0)

In node-based measures of semantic similarity, information content ($$IC$$) of a given term ($$c$$) can be computed as: $$IC(c) = -log~p(c)$$

Where $$p(c)$$ is the probability of occurrence of term $$c$$ in a corpus.

This method, described by Resnik, has been applied to biomedical ontologies. In this context, investigators can calculate $$p(c)$$ as the the probability of occurrence of a term $$c$$ being annotated. Although $$c$$ may exist in the corpus, it's possible that $$c$$ has $$0$$ annotations (ex. a gene with unknown function -- gene is in corpus, but unannotated).

Assuming $$p(c)$$ is the probability a term is annotated in a corpus:

$$p(c_1) = 0.50$$

$$p(c_2) = 0.10$$

$$p(c_3) = 0.00$$

Yields the following $$IC$$:

$$ICc_1 = 0.693$$

$$ICc_2 = 2.303$$

$$ICc_3 = undefined$$

The specific problem of $$ICc_3 = undefined$$ has been approached by maximizing $$IC$$ for unannotated terms and documented here. Excerpt: Alternatively, I've seen authors set $$IC=0$$ for unannotated terms, one example, Excerpt: Can anyone explain the motivation of the authors setting $$IC = 0$$ for terms that are unannotated in the second link? I'm struggling with this sharp contrast against the intuition that infrequently annotated terms should have higher $$IC$$. Or perhaps just shed some light on the fundamental differences and how to best avoid $$-log(0)$$?

# Edit:

Batet, et al. propose an alternative approach to calculating information content where "1 is added to the numerator and denominator to avoid log(0)".

The simplest most common way to avoid a 0 probability in word frequencies is the Lidstone smoothing

Which is basically, instead of using $$p(w_i)=\frac{\#(w_i)}{\sum{\#(w_j)}}$$ Use: $$p(w_i)=\frac{\#(w_i)+\epsilon}{\sum{\#(w_j)}+N\epsilon}$$

Regarding information of $p=0-$ The motivation I know is taken from the entropy definition: $$H(p)=p\log{p}$$ And when $$p\rightarrow0$$ We can prove that $$H(p)\rightarrow0$$ Using l'hopital rule

• Thanks for the link & summary. Smoothing seems to be generally applicable, and more similar to the first authors' method. Oct 25, 2016 at 15:04

I think the appropriate answer will depend on what you want to use the "information content" for in the end.

I personally do not like the justification given for the second case, i.e. "R gives log(0)=NaN". Mathematically we would have $$\mathrm{IC}=-\log=\infty$$ so to me the more relevant consideration would be the reason that the term has $p=0$.

Why would this make sense? If a word is very common in the corpus ($p\approx{1}$) then intuitively it is not very informative ($\mathrm{IC}\approx{0}$), i.e. knowing that a term occurs in a document tells us very little about that document specifically. Conversely, if a term is very rare in the corpus ($p\ll{1}$) we might expect the term to be very informative ($\mathrm{IC}\gg{0}$), i.e. documents in the corpus that contain that term are "distinctive" somehow. Following this logic, then in the limit of $p=0$ we might say that if we see the missing term in a new document, we might say that document is "infinitely distinct" from the corpus.

Why might this be a problem? I can think of two cases:

• If a term is "expected" in the corpus (i.e. you are bothering to record a $p$ for it), then it could be missing due to the finite sample size. In this case your "$p=0$" is more of a "below measurement threshold", so really you have censored data, i.e. $p<\frac{1}{N}$ for an $N$ document corpus. In this case your information content is also censored, i.e. $\mathrm{IC}>\log{N}$. The first approach you mention seems to be based on similar reasoning (i.e. $\mathrm{IC}\equiv\max_{p>0}\mathrm{IC}[p]$).

or, alternatively:

• The term is so common that it would have $p=1$, but this was known beforehand, so it was simply excluded from the analysis (i.e. a stop word). This would be a rational basis for the second approach you mention (i.e. $\mathrm{IC}\equiv{0}$)
• Thanks, I agree, the second case mentioned above confuses me a bit. In the context of biomedical ontologies, the censored data approach makes more sense than the stop word rationale. Oct 25, 2016 at 15:12