1
$\begingroup$

It's well known that language can be modeled by Multinomial distribution and Multiple Bernoulli distribution.

So far I don't see any advantage of Multiple Bernoulli distribution representation over Multinomial representation.

Both models equal computationally, Multinomial considers the number of occurrence of the unigram/bigram/.. tehrefore it's more precise.

What's the case when Multiple Bernoulli is preferable over Multinomial.

$\endgroup$

1 Answer 1

1
$\begingroup$

I presume you came up with this question after looking at naive Bayes models for text classification so I will answers this questions as such.

Event model

I think the question of which event model to use, is very much domain dependent. In many settings it makes more sense to model the data as a series of binary coin flips, each independent of each other. For instance, this recent work on Big Data, shows a superiority of Bernouilli event models in the context of binary transactional data gather from the internet. As a second example, it turns out that when one wants to model human behaviour, models like the Wallenius distribution are better fit for the job.

As far as I know, the multinomial distribution is simply a better model for document classification tasks because it fits the generation process of the data better if you are using a term-frequency representation for your data. A change of representation would require a re-evaluation of the event model in use and perhaps, there a Bernouilli or Wallenius event model could prove to be better.

Computational remarks

Bernouilli and multinomial are not equally expensive, in a term-frequency representation with a lot of zeros in the matrix, a multinomial model will be substantially faster because it does not need to look at the negative information. They are only equally expensive when dealing with binary data.

$\endgroup$
2
  • $\begingroup$ Thank you very much for your answer. If I understood correctly, the only reason to use Bernoulli model is "A change of representation". Does it mean, the change of estimation, when the estimation was changed during classification and there is a need to recalculate the classification? $\endgroup$
    – user16168
    Commented Jan 24, 2014 at 15:28
  • $\begingroup$ What I meant to say was a change in the representation of the input data prior to running your algorithms. So if instead of 'term-frequency' data vectors, you want to use say some meta-tag that you have on the document as input to your classification algorithm, then a Bernouilli distribution might be a more suitable choice. Also, note that none of the suggested models can cope with a change in input data very well after building the model. $\endgroup$
    – ciri
    Commented Jan 24, 2014 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.