Measuring Information Content of unannotated terms in a corpus, avoiding -log(0) In node-based measures of semantic similarity, information content ($IC$) of a given term ($c$) can be computed as:
$$IC(c) = -log~p(c)$$
Where $p(c)$ is the probability of occurrence of term $c$ in a corpus.
This method, described by Resnik, has been applied to biomedical ontologies.  In this context, investigators can calculate $p(c)$ as the the probability of occurrence of a term $c$ being annotated.  Although $c$ may exist in the corpus, it's possible that $c$ has $0$ annotations (ex. a gene with unknown function -- gene is in corpus, but unannotated).
Assuming $p(c)$ is the probability a term is annotated in a corpus:
$p(c_1) = 0.50$
$p(c_2) = 0.10$
$p(c_3) = 0.00$
Yields the following $IC$:
$ICc_1 = 0.693$
$ICc_2 = 2.303$
$ICc_3 = undefined$
The specific problem of $ICc_3 = undefined$ has been approached by maximizing $IC$ for unannotated terms and documented here.  Excerpt:

Alternatively, I've seen authors set $IC=0$ for unannotated terms, one example, Excerpt:

Can anyone explain the motivation of the authors setting $IC = 0$ for terms that are unannotated in the second link?
I'm struggling with this sharp contrast against the intuition that infrequently annotated terms should have higher $IC$.  Or perhaps just shed some light on the fundamental differences and how to best avoid $-log(0)$?
Edit:
Batet, et al. propose an alternative approach to calculating information content where "1 is added to the numerator and denominator to avoid log(0)".
Link to .pdf
 A: The simplest most common way to avoid a 0 probability in word frequencies is the Lidstone smoothing
Which is basically, instead of using
$$p(w_i)=\frac{\#(w_i)}{\sum{\#(w_j)}}$$
Use:
$$p(w_i)=\frac{\#(w_i)+\epsilon}{\sum{\#(w_j)}+N\epsilon}$$
Regarding information of $p=0-$ The motivation I know is taken from the entropy definition:
$$H(p)=p\log{p}$$
And when $$p\rightarrow0$$
We can prove that $$H(p)\rightarrow0$$
Using l'hopital rule
A: I think the appropriate answer will depend on what you want to use the "information content" for in the end.
I personally do not like the justification given for the second case, i.e. "R gives log(0)=NaN". Mathematically we would have
$$\mathrm{IC}[0]=-\log[0]=\infty$$
so to me the more relevant consideration would be the reason that the term has $p=0$.
Why would this make sense? If a word is very common in the corpus ($p\approx{1}$) then intuitively it is not very informative ($\mathrm{IC}\approx{0}$), i.e. knowing that a term occurs in a document tells us very little about that document specifically. Conversely, if a term is very rare in the corpus ($p\ll{1}$) we might expect the term to be very informative ($\mathrm{IC}\gg{0}$), i.e. documents in the corpus that contain that term are "distinctive" somehow. Following this logic, then in the limit of $p=0$ we might say that if we see the missing term in a new document, we might say that document is "infinitely distinct" from the corpus.
Why might this be a problem? I can think of two cases:


*

*If a term is "expected" in the corpus (i.e. you are bothering to record a $p$ for it), then it could be missing due to the finite sample size. In this case your "$p=0$" is more of a "below measurement threshold", so really you have censored data, i.e. $p<\frac{1}{N}$ for an $N$ document corpus. In this case your information content is also censored, i.e. $\mathrm{IC}>\log{N}$. The first approach you mention seems to be based on similar reasoning (i.e. $\mathrm{IC}[0]\equiv\max_{p>0}\mathrm{IC}[p]$).


or, alternatively:


*

*The term is so common that it would have $p=1$, but this was known beforehand, so it was simply excluded from the analysis (i.e. a stop word). This would be a rational basis for the second approach you mention (i.e. $\mathrm{IC}[0]\equiv{0}$)

