A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

# Tag Info

17

You always need this 'fail-safe' probability. To see why consider the worst case where none of the words in the training sample appear in the test sentence. In this case, under your model we would conclude that the sentence is impossible but it clearly exists creating a contradiction. Another extreme example is the test sentence "Alex met Steve." where "...

15

Let's say you've trained your Naive Bayes Classifier on 2 classes, "Ham" and "Spam" (i.e. it classifies emails). For the sake of simplicity, we'll assume prior probabilities to be 50/50. Now let's say you have an email $(w_1, w_2,...,w_n)$ which your classifier rates very highly as "Ham", say $$P(Ham|w_1,w_2,...w_n) = .90$$ and $$P(Spam|w_1,w_2,..w_n) = .... 11 With prior \mathsf{Unif}(0,1) \equiv \mathsf{Beta}(\alpha_0=1,\beta_0 =1) and likelihood \mathsf{Binom}(n, \theta) showing x successes in n trials, the posterior distribution is \mathsf{Beta}(\alpha_n=1 + x,\; \beta_n = 1 + n - x). (This is easily seen by multiplying the kernels of the prior and likelihood to get the kernel of the posterior.) ... 10 Sure. This is essentially the observation that the Dirichlet distribution is a conjugate prior for the multinomial distribution. This means they have the same functional form. The article mentions it, but I'll just emphasize that this follows from the multinomial sampling model. So, getting down to it... The observation is about the posterior, so let's ... 10 This is called Laplace's smoothing, or Laplace's rule of succession, as Pierre-Simon Laplace used it for estimating the probability the sun rises again tomorrow: "We thus find that an event having occurred a number of times, the probability that it will happen again the next time is equal to this number increased by the unit, divided by the same number ... 9 I think you want to have a look at what the text-mining people call smoothing. A simple smoothing technique is to add one to every word count, so no word has a zero probability estimate - essentially pretend that every word occurs once more than it does in reality. Generalized, this is sometimes called "Laplace smoothing" or "additive smoothing" - it's a ... 7 Typically one would use Laplace smoothing, essentially adding an artificial observation of every feature for every class. This is done to avoid the issue of having never observed a feature in one class causing a zero that propagates. This is also called a uniform prior. For a feature never seen ever in any training data, the "uniform prior" means ... 7 It is hard to answer your needs without more detail. In text analysis, word frequencies are replaced by tf*idf which stands for "term frequency times inverse document frequency". This is an empirical score that corrects for the occurrence of terms that are frequent in the corpus and thus do not discriminate documents. It is widely used to compare texts, in ... 5 This question is rather simple if you are familiar with Bayes estimators, since it is the directly conclusion of Bayes estimator. In the Bayesian approach, parameters are considered to be a quantity whose variation can be described by a probability distribution(or prior distribution). So, if we view the procedure of picking up as multinomial distribution, ... 5 You could call it a shrinkage estimator. The estimator is closer to .5 than the more ubiquitous sample mean. 4 For these kind of questions, it is possible to use Laplace Smoothing. In general Laplace Smoothing can be written as:$$ \text{If } y \in \begin{Bmatrix} 1,2,...,k\end{Bmatrix} \text{then,}\\ P(y=j)=\frac{\sum_{i=1}^{m} L\begin{Bmatrix} y^{i}=j \end{Bmatrix} + 1}{m+k} $$Here L is the likelihood. So in this case the emission probability values ( b_i(o)... 4 You might see Jeff Simonoff's 1996 book "Smoothing Methods in Statistics" (Springer) that devotes a chapter to this subject: In effect, one 'shrinks' the empirical cell probabilities to the 'overall' or 'uniform' cell probabilities using kernel weighting for discrete outcomes. This is designed exactly for the sparse settings you envision. To avoid the zeros ... 4 Disregarding those words is another way to handle it. It corresponds to averaging (integrate out) over all missing variables. So the result is different. How? Assuming the notation used here:$$ P(C^{*}|d) = \arg\max_{C} \frac{\prod_{i}p(t_{i}|C)P(C)}{P(d)} \propto \arg\max_{C} \prod_{i}p(t_{i}|C)P(C) $$where t_{i} are the tokens in the vocabulary and d... 4 Laplace smoothing is a way to move probabilities towards uninformed mean. Suppose you have a multinomial variable with sample counts c_1, c_2,..,c_d, where d is the number of dimensions. A Laplace smoothed version of estimated probabilities has the form: (c_i + \alpha)/(N + d\alpha), where \alpha is positive. If \alpha is 0 then we have non ... 3 We do not use logarithms because summing them gives different results then multiplying the non-logs, but because they behave the same. \log 0 = \infty and x + \infty = \infty, so after taking logs you will still end up with zeros. The whole idea of Laplace smoothing is that you adjust your data so that zeros become some more or less arbitrary small ... 3 As mentioned in the documentation, you have to set type='raw' within predict. -type: If "raw", the conditional a-posterior probabilities for each class are returned, and the class with maximal probability else. About the needs for Laplace smoothing (from Wikipedia Naive Bayes paper): If a given class and feature value never occur together in ... 2 You'll probably need to do some kind of Bayesian smoothing on the probabilities in the transition matrix that you build, based upon the data. Just because you have never seen the sequence "abb acc abc" in your data set does not mean that it has probability 0. Indeed, in some sense, a probability of 0 is awfully unlikely (unless we have some prior knowledge)... 2 You want to know why we bother with smoothing at all in a Naive Bayes classifier (when we can throw away the unknown features instead). The answer to your question is: not all words have to be unknown in all classes. Say there are two classes M and N with features A, B and C, as follows: M: A=3, B=1, C=0 (In the class M, A appears 3 times and B only ... 2 The data samples suggest tennis would never be played on overcast days irrespective of the temperature, humidity and wind. Is this realistic or do we lack a sufficient number of samples? Specifically note that tennis is being played on rainy days (i.e. with clouds in the sky like on overcast days), also if it's cool, humidity is high and the wind is strong. ... 2 A value 1 is added to each feature count (not just to the feature having a count/frequency of 0 ). To be more specific, consider the case of text classification. Let us suppose that Y_k, k=1,2,\cdots K denote the labels of the K classes, X_j denote the j^{th} word and V denote the total number of distinct words(vocabulary size) in all of the n=\... 2 The correct equation for P(w|c) should instead be P(w|c)= \frac{count(w,c)+1}{count(c)+|V|} assuming that there are V words in class c. If you make this correction, all your probabilities add to 1, as desired. 2 If you check the source code, predict.naiveBayes method from e1071 package simply ignores the new columns from test set (words that were not seen in training set) and iterates over the ones that were seen in the training set. attribs <- match(names(objecttables), names(newdata)) [...] L <- sapply(1:nrow(newdata), function(i) { ndata <- newdata[... 1 Sorry this is certainly too late to help you, but answering in case others find the question. When using Laplace smoothing, you should apply the smoothing for all attributes. To see why this is important, consider a dataset containing exactly 1 instance of a certain attribute and 0 instances of another. If you only applied the smoothing factor to the ... 1 You can try a simple smoothing - if your predicted probability is Pi, and you know the prior (0.5) then Pi -> Pi*(1-alpha) + prior*alpha Where alpha is a smoothing parameter between 0 and 1. Similar to Laplace smoothing. 1 A couple of thoughts come to mind. First, I think Laplace Smoothing -- not to be confused with Laplacian Smoothing -- might be useful. Basically, you add a small value to every cell so there are no 0's. Second, (C/D)/(A/0) is actually zero: (C/D) * (0/A). But that's probably not helpful in this case. Still, a rearrangement of your algebra/comparison might ... 1 Actually the above answer is a little incorrect in that, when we are adding 1 to a zero element, we should also divide by P(Y)+1 so that would be: \frac{5}{14} \cdot \frac{0+1}{5+1} \cdot \frac{1+1}{5+1} \cdot \frac{4+1}{5+1} \cdot \frac{3+1}{5+1} = 0.011 1 EDIT Feb 2, 2019: I figured I made a mistake below with the continuation count parameter (and with how a wolf sounds). I corrected it but published a new version in this link. If you might find it useful, please follow it there. I got here willing to find an answer but got no luck :( But then, I had to figure things out on my own! So let me try to explain ... 1 It is a common problem that in natural language data you do not observe some values that possibly can occur (e.g. you count frequencies of letters and in your data some uncommon letter does not occur at all). In this case the classical estimator for probability, i.e.$$ \hat p = \frac{n_i}{\sum_i n_i}  where $n_i$ is a number of occurrences of $i$th ...

1

Short answer: although it's possible to use it in this strange way, Kneyser-Ney is not designed for smoothing unigrams, because in this case its nothing but additive smoothing: $p_{abs}\left ( w_{i} \right )=\frac{max\left ( c\left ( w_{i} \right )-\delta ,0 \right )}{\sum_{w'}^{ }c(w')}$. This looks similar to Laplace smoothing and it is very well-known ...

1

Since no one else has chimed in, here my preliminary, not super well developed thoughts on this. if $A$ and $B$ are independent, then $P(A|B)P(B) = P(A)P(B)$. Therefore, if we can't find are unable to reject the hypothesis that $A \perp B$, then we should expect that replacing the likelihood term $P(A|B)$ (let's call this the "conditional likelihood") with \$...

Only top voted, non community-wiki answers of a minimum length are eligible