All Questions
Tagged with language-models naive-bayes
3 questions
1
vote
0
answers
201
views
Language Identification Better Results with Unigrams
I have a school project which consists of identifying each language of a tweet from a dataset of tweets. The dataset contains tweets in Spanish, Portuguese, English, Basque, Galician and Catalan. The ...
user255085
1
vote
1
answer
4k
views
A simple numerical example for Kneser-Ney Smoothing
I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams:
$P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) -
\...
16
votes
3
answers
5k
views
In Kneser-Ney smoothing, how are unseen words handled?
From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as
$
\begin{align}
P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
- 311