So far i have evaluated mn Bayes and Bernoulli, so my question is if i take the counts of the words of each document and use them for assigning the document to the particular class will it work with Multivariate Gaussian classifier (Bayes with Gaussian model)?
0
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
it depends on the kinds of tests you work with. this picture from (http://m.technologyreview.com/view/520311/tweets-have-become-shorter-since-2009-say-computer-scientists/) shows the twitter message length distribution 
since the length is capped at 140, I doubt that Gaussian will work well here.
$\begingroup$
$\endgroup$
7
A Gaussian model for word counts wouldn't fit well because of the zero counts you'd almost certainly have.
If you had a very small vocabulary, then perhaps you could use the square root transformation and do well with the Gaussian as an approximation. But in reality, "as a document generally uses only a small subset of the entire dictionary of term generated for a given database, most of the elements of a term-by-document matrix are zero" (from Matrices, Vector Spaces, and Information Retrieval).
You can't transform your way out of zero! That's why exact distributions like multinomial are a better choice in word-count modeling situations like yours.
If you consider Wikipedia an "official" source, the page on the Normal Approximation should convince you that the conditions are not met for word counts. The probability is too small relative to the number of words; you're getting zeros most of the time.
-
1$\begingroup$ I don't think I agree with this. Gaussian's can have zeros. The trouble is that if your matrix is simply zeros and ones, you will probably want to use a Bernoulli. You can also use the Poisson distribution but the former is probably a better model if you know that 1 is really the max. If you just go ahead and use the Gaussian distribution, it should still work but might just give you identical results to the Bernouli. Sums of Bernoullis are Poisson distributed and for large numbers this becomes approximately Gaussian. $\endgroup$ Commented Mar 2, 2014 at 22:03
-
$\begingroup$ Dave, it sounds like we both agree that a discrete distribution would work better for modeling low-count data than the Gaussian. Sums of Bernoulli random variables are either Binomial (constant success probability) or Poisson Binomial (different success probabilities), and regarding the Gaussian approximation to these distributions, you are not going to have large enough numbers necessary for it to be adequate. You say that Gaussian random variables can be zero, but I assure you with probability 1 they will not be! $\endgroup$ Commented Mar 3, 2014 at 6:58
-
$\begingroup$ Ben, a Gaussian has non-zero probability from minus infinity to positive infinity regardless of the means and variance and so that includes zero. A standard normal has it's mean and mode at zero. $\endgroup$ Commented Mar 3, 2014 at 12:37
-
$\begingroup$ It does sound strange that the value corresponding to the highest probability density still has probability zero of occurring. But I'm not making this up. Search for the line "Probability of a single point is zero" in these class notes: courses.washington.edu/dphs568/course/day4.htm. It takes a while to wrap your head around. $\endgroup$ Commented Mar 4, 2014 at 5:10
-
2$\begingroup$ That's simply because of the fact that the Gaussian is a continuous distribution. It's the probability density at a point, not the probability OF a point that matters. One could also say that computers works with discrete floating point numbers not real numbers. You can still do regression on data that has been forced to be integers, even simply 0 or 1. The calculation just sums things anyway and so it is these sums that effect the result. The sum of many independent variables tends toward the Gaussian anyway regardless of their individual distribution (with some mild assumptions). $\endgroup$ Commented Mar 4, 2014 at 13:23