Questions tagged [language-models]
A statistical language model is a probability distribution over sequences of words.
110 questions
1
vote
1
answer
511
views
Curse of dimensionality with language models
In the seminal paper A Neural Probabilistic Language Model, Yoshua Bengio and his colleagues make the following point:
If one wants to model the joint probability distribution of 10
consecutive ...
- 6,217
2
votes
1
answer
145
views
How many words does the algorithm search through in Google Ngram? [closed]
When I run a query for "hers" in Google Ngram Viewer, I get back the word's frequency of occurrence as a percentage. We know the outcome percentage; what's the denominator on the other size? Is it 100 ...
- 646
16
votes
1
answer
10k
views
What are the pros and cons of applying pointwise mutual information on a word cooccurrence matrix before SVD?
One way to generate word embeddings is as follows (mirror):
Get a corpora, e.g. "I enjoy flying. I like NLP. I like deep learning."
Build the word cooccurrence matrix from it:
Perform SVD on $X$, ...
- 47.6k
2
votes
1
answer
937
views
Why two layers of matrix multiplication is needed in CBOW and Skip-gram model?
I found a nice tutorial here regarding CBOW and Skip-gram models for Word2Vec. I got the following doubt:
Why two layers of matrix multiplication is need for CBOW and SKIP-GRAM ? As multiplying a ...
- 153
11
votes
2
answers
3k
views
Question about Continuous Bag of Words
I'm having trouble understanding this sentence:
The first proposed architecture is similar to the feedforward NNLM,
where the non-linear hidden layer is removed and the projection
layer is ...
- 323
0
votes
0
answers
134
views
How to tell if the occurence of a variable is statistically significant/meaningful?
Since my previous question seems to be unclear, I will try to put it in other words.
I have been working on a child speech corpus, which consists of sentences in which children of various ages ...
- 101
16
votes
3
answers
5k
views
In Kneser-Ney smoothing, how are unseen words handled?
From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as
$
\begin{align}
P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
- 311
7
votes
2
answers
6k
views
Does trigram guarantee to perform more accurately than bigram?
When implementing some NLP project, such as text segmentation, Name Entity Recognition, does using trigram guarantee to perform more accurately than bigram?
$$
Trigram: p(s_t\mid s_{t-2}, s_{t-1})
$$...
- 405
8
votes
1
answer
268
views
Language modeling: why is adding up to 1 so important?
In many natural language processing applications such as spelling correction, machine translation and speech recognition, we use language models. Language models are created usually by counting how ...
- 183
11
votes
3
answers
16k
views
Regarding using bigram (N-gram) model to build feature vector for text document
A traditional approach of feature construction for text mining is bag-of-words approach, and can be enhanced using tf-idf for setting up the feature vector characterizing a given text document. At ...
- 3,089