What is the difference between these two probabilities?

Question

I am working on a word sense disambiguation task and I have a dataset consisting of labelled sentences for two meanings of the word 'line'. I am trying to find 'trigger' words that are good at determining the sense of the ambiguous word if it occurs with it in the sentence. So, for each word in my training data (except the ambiguous word 'line') I am trying to calculate the quantities

P(sense_1 | word) and P(sense_2 | word)

I can think of two ways of calculating this and both make sense to me. The first is

P(sense_i | word)=count(word,sense_i)/count(word)

but then there is also Bayes Rule

P(sense_i|word)=P(sense_i)P(word|sense_i)=[count(sense_i)/total_examples] * [count(word,sense_i)/[sum_w count(w,sense_i)]]

where first equality is intended to mean proportional.

I do not think that they are equivalent, so is the math wrong for one of them?

Answer 1

The first approach seems right whereas in the Bayes rule approach a factor got lost: $$P(\text{sense_i}|\text{word})P(\text{word}) = P(\text{word}|\text{sense_i})P(\text{sense_i})$$ Hence $$P(\text{sense_i}|\text{word})= \frac{P(\text{word}|\text{sense_i})P(\text{sense_i})}{P(\text{word})}\\ $$ Inserting the counts yields $$\frac{\frac{\text{count(word,sense_i)}}{\text{count(sense_i)}} \times \frac{\text{count(sense_i)}}{\text{count(num.examples)}}}{\frac{\text{count(word)}}{\text{count(num.examples)}}}= \frac{\text{count(word,sense_i)}}{\text{count(word)}}$$ which is exactly the same result as in the first approach.

EDIT #1: It is important to realize that $P(\text{sense_i}|\text{word})$ is only conditional on a single word. Therefore, at the end of the day only sentences containing this word can contribute information to calculate this probability.

EDIT #2: One can express $\text{count(sense_i)}$ as a sum, i.e. $$\text{count(sense_i)} = \text{count(}\neg\text{word,sense_i}) + \text{count(word,sense_i)}$$ Other possible words are not represented in this sum. In essence $\text{word}$ represents a random variable that can take two possible states: the particular word is present in the sentence or it is not. For every possible word appearing with the word line such a variable is defined. This is something different than having a variable $\text{word}$ which could take every occurent word as value. Think about the meaning of such a variable. If more than one word occurs in a sentence which word should be the value of $\text{word}$?

Maybe it also helps to think about the way to calculate the probability that line has $\text{sense_i}$ in a particular sentence. To make use of every conditional probability $P(\text{word}_1|\text{sense_i})$, $P(\text{word}_2|\text{sense_i})$, $\dots$ in deciding in which meaning line is used, an additional assumption is necessary. If one assumes that the incidence of one word is independent of the incidence of another one, i.e. $$\eqalign{P(\text{word}_1|\text{word}_2)&=P(\text{word}_1) \\ P(\text{word}_1,\text{word}_2)&=P(\text{word}_1)P(\text{word}_2)}$$ then the calculation can be performed by $$P(\text{sense_i}|\text{word}_1,\text{word}_2,\dots) = \frac{P(\text{word}_1|\text{sense_i})P(\text{word}_2|\text{sense_i})\dots} {P(\text{word}_1)P(\text{word}_2)\dots}P(\text{sense_i})$$ with $$\eqalign{P(\text{word}_{\,i})&=\frac{\text{count(word}_{\,i}\text{)}}{\text{count(num.examples)}}\\ P(\text{word}_{\,i}|\text{sense_i})&=\frac{\text{count(word}_{\,i}\text{,sense_i})}{\text{count(sense_i)}}}$$