1
$\begingroup$

I am reading Klein and Manning's notes on Maximum Entropy for Natural Language Processing. On slide 22, they have an equation saying, $P(C|D,\lambda) = \Pi _{(c,d)\in (C,D)} P(c|d,\lambda)$. I am not sure how they obtain this or what it really means. In the coursera video on this topic, Manning says that this follows because the random variables are independent and identically distributed. However, I thought, Maximum Entropy methods are prefered over Naive Bayes because they don't make the "naive" independence assumptions. If someone has a reference for this equation, that will be great.

$\endgroup$
1
$\begingroup$

The individual samples in the dataset are iid. The features within a given sample are not assumed to be iid.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the answer. I am still a bit confused though. What does C really mean if we are varying over the samples in a dataset? For example, if are doing named entity recognition (an example that they have been using), the individual date pieces (words in this instance) within a document can be associated with a class and a corresponding feature weight. What does $P(C|D,\lambda)$ mean in this case? $\endgroup$ – user21338 Feb 27 '13 at 18:08
  • $\begingroup$ P(C|D,λ) refers to the probability distribution of classes to every instance in the dataset, given a particular weight vector λ. EG, the probability that instance 1 has class 3 and instance 2 has class 1 and instance 3 has class 1 is a single entry in that distribution. If there are m classes and n instances, there are m^n entries in the distribution. $\endgroup$ – rrenaud Feb 28 '13 at 0:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy