Questions tagged [language-models]
A statistical language model is a probability distribution over sequences of words.
110 questions
16
votes
1
answer
10k
views
What are the pros and cons of applying pointwise mutual information on a word cooccurrence matrix before SVD?
One way to generate word embeddings is as follows (mirror):
Get a corpora, e.g. "I enjoy flying. I like NLP. I like deep learning."
Build the word cooccurrence matrix from it:
Perform SVD on $X$, ...
- 47.6k
16
votes
3
answers
5k
views
In Kneser-Ney smoothing, how are unseen words handled?
From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as
$
\begin{align}
P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
- 311
11
votes
2
answers
3k
views
Question about Continuous Bag of Words
I'm having trouble understanding this sentence:
The first proposed architecture is similar to the feedforward NNLM,
where the non-linear hidden layer is removed and the projection
layer is ...
- 323
11
votes
3
answers
16k
views
Regarding using bigram (N-gram) model to build feature vector for text document
A traditional approach of feature construction for text mining is bag-of-words approach, and can be enhanced using tf-idf for setting up the feature vector characterizing a given text document. At ...
- 3,089
8
votes
1
answer
2k
views
Why can't standard conditional language models be trained left-to-right *and* right-to-left?
From the BERT paper:
Unfortunately, standard conditional language models can only be trained left-to-right or right-to-left, since bidirectional conditioning would allow each word to indirectly “...
- 1,386
8
votes
1
answer
573
views
Why are Transformers "suboptimal" for language modeling but not for translation?
Language Models with Transformers states:
Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to ...
- 1,347
8
votes
1
answer
268
views
Language modeling: why is adding up to 1 so important?
In many natural language processing applications such as spelling correction, machine translation and speech recognition, we use language models. Language models are created usually by counting how ...
- 183
7
votes
3
answers
2k
views
How do you add knowledge to LLMs?
I recently heard an interesting comment from a gentleman on YouTube and it made sense instantly.
To paraphrase he explained that "fine-tuning" an LM is not necessarily adding knowledge to a ...
- 768
7
votes
1
answer
2k
views
How does one design a custom loss function? What features make a loss function "good"?
I have a custom situation for which I am trying to design a cost function.
The idea is that you have a stack of LSTMs doing something slightly unconventional. Each LSTM$_l$ computes a linear ...
- 277
7
votes
2
answers
2k
views
Why are language modeling pre-training objectives considered unsupervised?
Maybe this is stemming from my not-so-great grasp of supervised vs. unsupervised learning, but my understanding is that if we have access to ground-truth labels then it's supervised learning and if ...
- 4,177
7
votes
2
answers
1k
views
Neural network language model - prediction for the word at the center or the right of context words
Neural network language model - prediction for the word at the center or the right of context words?
On Bengio's paper, the model predicts probability by n words for the next word, like predicting ...
- 848
7
votes
2
answers
3k
views
Calculating test-time perplexity for seq2seq (RNN) language models
To compute the perplexity of a language model (LM) on a test sentence $s=w_1,\dots,w_n$ we need to compute all next-word predictions $P(w_1), P(w_2|w_1),\dots,P(w_n|w_1,\dots,w_{n-1})$.
My question ...
- 96
7
votes
2
answers
6k
views
Does trigram guarantee to perform more accurately than bigram?
When implementing some NLP project, such as text segmentation, Name Entity Recognition, does using trigram guarantee to perform more accurately than bigram?
$$
Trigram: p(s_t\mid s_{t-2}, s_{t-1})
$$...
- 405
6
votes
1
answer
525
views
n-gram language model
At the end of the introduction of A Neural Probabilistic Language Model (Bengio et al. 2003), the following example is given:
Having seen the sentence ...
- 6,217
5
votes
2
answers
1k
views
Generating text from language model
I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be:
Apply softmax function
Take a weighted random choice to determine next word
This ...
5
votes
1
answer
1k
views
Skip-gram algorithm confusion
As a newbie to NLP, I am (deeply) confused by the middle step in the following diagram explaining the skip-gram algorithm. The video where this diagram was presented can be found at:
https://www....
- 377
5
votes
2
answers
12k
views
Understanding Add-1/Laplace smoothing with bigrams
I am working through an example of Add-1 smoothing in the context of NLP
Say that there is the following corpus (start and end tokens included)
...
- 173
5
votes
1
answer
1k
views
Should a language model like GPT-3 be directly used to perform classification?
The OpenAI API enables classification by sampling from GPT-3 given a prompt. Is estimating posterior probabilities a more statistically sound approach? Below is a specification of what "...
- 1,625
5
votes
1
answer
8k
views
Advantage of character based language models over word based
Is there an intuition why character based models language bases models are preferred over word based. For example Karpathy builds his language model by predicting the next character in Karpathy Blog.
...
- 211
5
votes
1
answer
1k
views
Denoising autoencoders vs masked approaches
I am not an expert in language modelling domain.
There are mainly two approaches that are being used nowadays. Denoising autoencoders and ...
- 378
4
votes
1
answer
3k
views
Language Model compare probability scores between Length varying sentence
My question is : How can I compare Language Model(LM) score for two sentences with different lengths ?
Probabilities are < 1...
3
votes
2
answers
2k
views
Language Detection with CLD2 with Mixed Inputs in long documents
Internals Recap. CLD2 is a Naïve Bayesian classifier, trained on documents of mean size of 200 characters, trained on a corpus of 100M scraped and human expert selected web pages.
When working on ...
- 153
3
votes
1
answer
4k
views
Use of ignore_index on CrossEntropyLoss() for text models
I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this.
From what I understand ...
- 33
3
votes
1
answer
187
views
Breaking substitution cipher with language model
Frequency analysis is a common tool used to break substitution ciphers, but often relies on intuition and guesswork of a human. Since language models can objectively calculate perplexity (how ...
3
votes
2
answers
453
views
Language Modelling using Neural Networks
I plan to make a Language Model in Python using Neural Networks. I've read that Neural Networks need vectors as input. One common vector representation in NLP is the Bag of Words model. Given a corpus ...
- 53
3
votes
1
answer
4k
views
How to handle big vocabulary size with keras tokenizer?
I am actually working on a neural language model developed with keras.
I have an encoder and a decoder and the output of the decoder is a dense vector on the vocabulary..so quite big depending on the ...
- 31
3
votes
0
answers
1k
views
Different ways to calculate pointwise mutual information for word co-occurrence [closed]
I have a (very) small corpus of documents. As a representative example: 450 documents, 280000 total word count.
I am calculating Positive Pointwise Mutual Information (PPMI) between a selection of ...
- 133
3
votes
0
answers
39
views
natural language understanding algorithms [closed]
I am doing some research into how smart personal assistants work like siri, alexa etc. I have found that it is using automatic speech recognition to turn the speech into weighted text form and then ...
- 31
3
votes
1
answer
220
views
How do we pass data to a RNN?
Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we:
Scheme 1
Take the first batch of ...
- 2,683
2
votes
1
answer
145
views
How many words does the algorithm search through in Google Ngram? [closed]
When I run a query for "hers" in Google Ngram Viewer, I get back the word's frequency of occurrence as a percentage. We know the outcome percentage; what's the denominator on the other size? Is it 100 ...
- 646
2
votes
1
answer
68
views
Machine Translation: with sufficient parallel data, can we improve even further the performance of the system with the use of monolingual data?
I am trying to find scientific literature that studies if, in a situation in which we already have enough parallel data, the addition of monolingual data can further improve performance.
I have not ...
- 23
2
votes
2
answers
120
views
What are commonly used methods to represent a document by a vector?
Methods that I know of
Bag of words + weighting: tf-idf, bm25
Topic models: LSA, LDA
Word/sentence/document embedding
Are there other commonly used methods to represent a document by a vector?
- 16.8k
2
votes
1
answer
937
views
Why two layers of matrix multiplication is needed in CBOW and Skip-gram model?
I found a nice tutorial here regarding CBOW and Skip-gram models for Word2Vec. I got the following doubt:
Why two layers of matrix multiplication is need for CBOW and SKIP-GRAM ? As multiplying a ...
- 153
2
votes
1
answer
220
views
multiple likely ys for one instance of x: word prediction with LSTM
I have a ML project that is about predicting (suggesting) the next word based on the last n words, using LSTM. The output is a softmax dense layer the size of the vocabulary that shows the probability ...
- 51
2
votes
1
answer
349
views
Does the transformers model (in “Attention is All You Need”) exclude the encoder in language modelling tasks?
The language model I am referring to is the one outlined in "Attention is All You Need":
My understanding is (please correct me if I am wrong) that when the task is translation, the encoder'...
- 23
2
votes
2
answers
1k
views
Calculating perplexity with smoothing techniques (NLP)
This question is about smoothed n-gram language models. When we use additive smoothing on the train set to determine the conditional probabilities, and calculate the perplexity of train data, where ...
- 23
2
votes
1
answer
515
views
What's the best practice for dealing with OOV characters?
I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
- 33
2
votes
3
answers
790
views
Confusion about CBOW and Skip-Gram models?
I've read a couple online description of CBOW and Skip-Gram and usually the descriptions starts like this:
We need to train models on words
So we encode words using vectors
One-hot encoding is not ...
- 1,630
2
votes
1
answer
730
views
How to sample a language model?
I've successfully trained a language model using LSTMs. But I have a confusion about sampling.
On sampling, we generate a probability distribution at each time step. It will be of length vocabulary ...
- 123
2
votes
1
answer
88
views
Finding a mixture of 1st and 0'th order Markov models that is closest to an empirical distribution
I am interested in finding the distribution "$p^*$" closest to an empirical distribution $\hat{p}$ where $p^*$ is a mixture of first and zeroth order Markov models. That is, I want to find $$
p^* = \...
- 751
2
votes
1
answer
1k
views
Bayesian smoothing using Dirichlet prior : why not MAP?
I am reading about smoothing methods for language model ( I am working on unigram model).
If you are not familiar with unigram model, it is closely related to multinomial distribution (with the ...
- 57
2
votes
2
answers
442
views
Language model created with SRILM does not sum to 1
I created an n-gram language model on the Penn Treebank using the following command:
...
- 21
2
votes
1
answer
37
views
Probability Distribution Underlying Ngram Model
Texts introducing ngram models often directly manipulate conditional probabilities. For example, given a corpus $V$ with a bigram model on its words, we would compute the probability of a sentence $...
- 73
2
votes
1
answer
91
views
What's the loss that is optimized in InstructGPT RL stage?
In the InstructGPT paper they define objective of RL stage as:
They try to maximize this objective using PPO.
I have trouble understanding how they plug this objective into the PPO though. Do they ...
- 143
2
votes
1
answer
403
views
Do I need training data in multiple languages for a multilingual transformer?
I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.
...
- 465
2
votes
1
answer
46
views
Language model gives different results when Bayes' theorem is applied
Say, the following example is our corpus:
a quick brown fox jumps over the lazy dog.
Here, there are 9 tokens in the corpus. ...
- 235
2
votes
1
answer
982
views
For BERT-based models, do we absolutely have to include the [CLS] and [SEP] special tokens in the input data?
The thought just occurred to me while I was processing data. If we're using the [CLS] token for classification, then it would obviously make sense to include it, ...
- 4,177
2
votes
1
answer
830
views
BERT masking - why does it require sampling, and how does it mitigate the mismatch of the [MASK] token when fine-tuning?
I'm reading the BERT paper and jalammar's illustrative guide for BERT.
I don't understand 2 things about the method's crux - the masked language model:
why does masking requires us to sample (take ...
- 3,360
2
votes
1
answer
189
views
How to compute context-independent token representations in a biLM?
I've been reading this paper on ELMo word representations.
For context, here's my understanding of the standard bi-directional language model (biLM) thus far:
Given a sequence of tokens $(t_{1}, ...
2
votes
1
answer
2k
views
what are hidden states in HMM based language model?
There are several ways to build language models, n-gram based models are straightforward, but for the language models built on HMMs, what are hidden states and what are observations?
- 33