Measuring Information Content of unannotated terms in a corpus, avoiding -log(0)

Question

In node-based measures of semantic similarity, information content ($IC$) of a given term ($c$) can be computed as: $$IC(c) = -log~p(c)$$

Where $p(c)$ is the probability of occurrence of term $c$ in a corpus.

This method, described by Resnik, has been applied to biomedical ontologies. In this context, investigators can calculate $p(c)$ as the the probability of occurrence of a term $c$ being annotated. Although $c$ may exist in the corpus, it's possible that $c$ has $0$ annotations (ex. a gene with unknown function -- gene is in corpus, but unannotated).

Assuming $p(c)$ is the probability a term is annotated in a corpus:

$p(c_1) = 0.50$

$p(c_2) = 0.10$

$p(c_3) = 0.00$

Yields the following $IC$:

$ICc_1 = 0.693$

$ICc_2 = 2.303$

$ICc_3 = undefined$

The specific problem of $ICc_3 = undefined$ has been approached by maximizing $IC$ for unannotated terms and documented here. Excerpt:

Alternatively, I've seen authors set $IC=0$ for unannotated terms, one example, Excerpt:

Can anyone explain the motivation of the authors setting $IC = 0$ for terms that are unannotated in the second link? I'm struggling with this sharp contrast against the intuition that infrequently annotated terms should have higher $IC$. Or perhaps just shed some light on the fundamental differences and how to best avoid $-log(0)$?

Edit:

Batet, et al. propose an alternative approach to calculating information content where "1 is added to the numerator and denominator to avoid log(0)".
Link to .pdf

Answer 1

The simplest most common way to avoid a 0 probability in word frequencies is the Lidstone smoothing

Which is basically, instead of using $$p(w_i)=\frac{\#(w_i)}{\sum{\#(w_j)}}$$ Use: $$p(w_i)=\frac{\#(w_i)+\epsilon}{\sum{\#(w_j)}+N\epsilon}$$

Regarding information of $p=0-$ The motivation I know is taken from the entropy definition: $$H(p)=p\log{p}$$ And when $$p\rightarrow0$$ We can prove that $$H(p)\rightarrow0$$ Using l'hopital rule

Answer 2

I think the appropriate answer will depend on what you want to use the "information content" for in the end.

I personally do not like the justification given for the second case, i.e. "R gives log(0)=NaN". Mathematically we would have $$\mathrm{IC}[0]=-\log[0]=\infty$$ so to me the more relevant consideration would be the reason that the term has $p=0$.

Why would this make sense? If a word is very common in the corpus ($p\approx{1}$) then intuitively it is not very informative ($\mathrm{IC}\approx{0}$), i.e. knowing that a term occurs in a document tells us very little about that document specifically. Conversely, if a term is very rare in the corpus ($p\ll{1}$) we might expect the term to be very informative ($\mathrm{IC}\gg{0}$), i.e. documents in the corpus that contain that term are "distinctive" somehow. Following this logic, then in the limit of $p=0$ we might say that if we see the missing term in a new document, we might say that document is "infinitely distinct" from the corpus.

Why might this be a problem? I can think of two cases:

• If a term is "expected" in the corpus (i.e. you are bothering to record a $p$ for it), then it could be missing due to the finite sample size. In this case your "$p=0$" is more of a "below measurement threshold", so really you have censored data, i.e. $p<\frac{1}{N}$ for an $N$ document corpus. In this case your information content is also censored, i.e. $\mathrm{IC}>\log{N}$. The first approach you mention seems to be based on similar reasoning (i.e. $\mathrm{IC}[0]\equiv\max_{p>0}\mathrm{IC}[p]$).

or, alternatively:

• The term is so common that it would have $p=1$, but this was known beforehand, so it was simply excluded from the analysis (i.e. a stop word). This would be a rational basis for the second approach you mention (i.e. $\mathrm{IC}[0]\equiv{0}$)