Questions tagged [language-models]
A statistical language model is a probability distribution over sequences of words.
110 questions
2
votes
2
answers
44
views
End-Tokens are Required to make Ngram Models Proper
The standard bigram model, (for example defined here) defines a probability distribution over a corpus $V$ based on the following principles:
The marginal probability of a word $w$ is defined as its ...
- 73
2
votes
1
answer
37
views
Probability Distribution Underlying Ngram Model
Texts introducing ngram models often directly manipulate conditional probabilities. For example, given a corpus $V$ with a bigram model on its words, we would compute the probability of a sentence $...
- 73
0
votes
0
answers
10
views
Ngram - Good Smoothing Probability Problem
I am working on creating an n-gram model for unigrams, bigrams, and trigrams. However, I have two questions:
Should the sum of the unseen probability and the observed probability always equal 1?
If 𝑁�...
- 1
3
votes
1
answer
220
views
How do we pass data to a RNN?
Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we:
Scheme 1
Take the first batch of ...
CommunityBot
- 1
0
votes
1
answer
3k
views
Finding the perplexity of multiple examples
I am trying to find a way to calculate perplexity of a language model of multiple 3-word examples from my test set, or perplexity of the corpus of the test set. As the test set, I have a paragraph ...
CommunityBot
- 1
1
vote
1
answer
340
views
Feedforward Neural Net Language Model - computational complexity (word2vec)
While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM):
It consists of input, projection, hidden and output layers. At the ...
CommunityBot
- 1
7
votes
3
answers
2k
views
How do you add knowledge to LLMs?
I recently heard an interesting comment from a gentleman on YouTube and it made sense instantly.
To paraphrase he explained that "fine-tuning" an LM is not necessarily adding knowledge to a ...
- 47.6k
0
votes
0
answers
12
views
Validation accuracy dip and recovery when restarting training
i was fine-tuning this large language model with Stochastic Gradient Descent and mid epoch i stopped training, and saved the model weights. Then at a later time, reloaded the weights and restarted the ...
- 348
1
vote
1
answer
91
views
How can we implement pipelines using LLMs that can consider information from the internet in their responses?
Let us suppose that we are implementing a data processing pipeline based on LLMs and, in some part of the process, we need to search the internet for finding relevant information regarding some topic, ...
- 46
2
votes
1
answer
91
views
What's the loss that is optimized in InstructGPT RL stage?
In the InstructGPT paper they define objective of RL stage as:
They try to maximize this objective using PPO.
I have trouble understanding how they plug this objective into the PPO though. Do they ...
- 1,210
0
votes
0
answers
248
views
Is it necessary to use attention mask for mean pooling for BERT?
I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
- 138
0
votes
0
answers
20
views
Why doesn't BERT give me back my original sentence?
I've started playing with BERT encoder through the huggingface module.
I passed it a normal unmasked sentence and got the ...
- 641
7
votes
2
answers
2k
views
Why are language modeling pre-training objectives considered unsupervised?
Maybe this is stemming from my not-so-great grasp of supervised vs. unsupervised learning, but my understanding is that if we have access to ground-truth labels then it's supervised learning and if ...
- 101
0
votes
0
answers
296
views
Bert Used for generative AI
I have a doubt regarding using "Bert" as a generative model. I know Bert can be used for classification or fine-tuning the question-answering. However, is it possible to use Bert to generate ...
- 175
5
votes
1
answer
1k
views
Should a language model like GPT-3 be directly used to perform classification?
The OpenAI API enables classification by sampling from GPT-3 given a prompt. Is estimating posterior probabilities a more statistically sound approach? Below is a specification of what "...
- 1,625
0
votes
0
answers
47
views
Classification in BERT - why not use class as a feature?
I am currently following this post, which details how BERT was trained. I had a few questions about the classification task:
In the post, it mentions that the authors of BERT decided to add ...
- 319
3
votes
0
answers
1k
views
Different ways to calculate pointwise mutual information for word co-occurrence [closed]
I have a (very) small corpus of documents. As a representative example: 450 documents, 280000 total word count.
I am calculating Positive Pointwise Mutual Information (PPMI) between a selection of ...
- 9,001
1
vote
0
answers
38
views
Language Model dealing with Book Dialogue
My goal is to generate text based on a specific book using an LSTM language model. One problem with my generator is that it seems that the book's dialogue is somewhat messing up my generator. I had ...
- 82.7k
1
vote
0
answers
91
views
Why does the best performing adapter-based parameter-efficient fine-tuning depend on the language model being fine-tuned?
https://arxiv.org/abs/2304.01933 shows that the best performing adapter-based parameter-efficient fine-tuning depends on the language model being fine-tuned:
E.g., LORA is the best adapter for LlaMa-...
- 47.6k
1
vote
1
answer
94
views
Intuitive difference between NN and attention for text prediction
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:
Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
- 141k
0
votes
1
answer
14
views
Continuous Bag of Words derivation
The continuous bag of words model has the following log probability for observing a sequence of words: $$\log P(\textbf{w})=\sum_{c=1}^{C}\log{P(w_c|w_{c-m},...w_{c-1}, w_{c+1},...,w_{c+m}})$$
I don't ...
- 3,459
0
votes
1
answer
34
views
Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)
I understand how later neural language models (such as those used in the Word2Vec papers) framed the language modelling problem in a self-supervised way by learning to predict the next word (or any ...
- 3,459
0
votes
1
answer
2k
views
Difference between "transfer learning" and "fine-tuning"
I am currently reading the BERT paper and it splits the use of pre-trained models into two categories:
Feature-based whereby you just take the embeddings for the tokens and plug it into whatever ...
- 3,459
0
votes
1
answer
484
views
Perplexity for different n-gram models
I'm training a Lidstone Model with different n-gram sequences to see witch one is the best (2-gram, 3-gram, 4-gram, etc) in the same text database.
When I give all these models an unseen text sample ...
- 1,625
8
votes
1
answer
573
views
Why are Transformers "suboptimal" for language modeling but not for translation?
Language Models with Transformers states:
Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to ...
- 1,347
2
votes
2
answers
1k
views
Calculating perplexity with smoothing techniques (NLP)
This question is about smoothed n-gram language models. When we use additive smoothing on the train set to determine the conditional probabilities, and calculate the perplexity of train data, where ...
- 1
2
votes
1
answer
403
views
Do I need training data in multiple languages for a multilingual transformer?
I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.
...
- 3,459
16
votes
3
answers
5k
views
In Kneser-Ney smoothing, how are unseen words handled?
From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as
$
\begin{align}
P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
- 9,001
1
vote
1
answer
468
views
How to handle out of the bag token in NLP?
In my current language model my model is unaware of any token that is out-of-bag
for example:- In my summary generating model when we pass some token that is out-of-bag then my model will completely ...
- 141k
1
vote
0
answers
91
views
What's the role of masking in transformer and BERT?
I've recently implemented the architectures of Transformer and BERT and found that they both have common property - masked layers among one of them. I have come to questions like below.
As far as I ...
- 94k
1
vote
0
answers
1k
views
Inference time for text genration using fine-tuned gpt2
I have re-trained GPT2 model using over 10 million sentences for QA. And while testing also I am getting very good results. The only problem now I am facing is that I have millions of test data that I ...
- 1
2
votes
1
answer
46
views
Language model gives different results when Bayes' theorem is applied
Say, the following example is our corpus:
a quick brown fox jumps over the lazy dog.
Here, there are 9 tokens in the corpus. ...
- 9,001
1
vote
3
answers
438
views
How to improve language model ex: BERT on unseen text in training?
so I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure.
However, when I remove a ...
- 1,930
1
vote
0
answers
2k
views
Which dimensionality reduction technique works well for BERT sentence embeddings?
I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
CommunityBot
- 1
2
votes
0
answers
622
views
Variable batch size for inputs of different length
We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
2
votes
1
answer
220
views
multiple likely ys for one instance of x: word prediction with LSTM
I have a ML project that is about predicting (suggesting) the next word based on the last n words, using LSTM. The output is a softmax dense layer the size of the vocabulary that shows the probability ...
- 141k
7
votes
1
answer
2k
views
How does one design a custom loss function? What features make a loss function "good"?
I have a custom situation for which I am trying to design a cost function.
The idea is that you have a stack of LSTMs doing something slightly unconventional. Each LSTM$_l$ computes a linear ...
- 94k
2
votes
1
answer
349
views
Does the transformers model (in “Attention is All You Need”) exclude the encoder in language modelling tasks?
The language model I am referring to is the one outlined in "Attention is All You Need":
My understanding is (please correct me if I am wrong) that when the task is translation, the encoder'...
- 9,001
2
votes
2
answers
442
views
Language model created with SRILM does not sum to 1
I created an n-gram language model on the Penn Treebank using the following command:
...
CommunityBot
- 1
2
votes
1
answer
982
views
For BERT-based models, do we absolutely have to include the [CLS] and [SEP] special tokens in the input data?
The thought just occurred to me while I was processing data. If we're using the [CLS] token for classification, then it would obviously make sense to include it, ...
- 3,459
3
votes
1
answer
4k
views
Use of ignore_index on CrossEntropyLoss() for text models
I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this.
From what I understand ...
- 9,001
7
votes
2
answers
6k
views
Does trigram guarantee to perform more accurately than bigram?
When implementing some NLP project, such as text segmentation, Name Entity Recognition, does using trigram guarantee to perform more accurately than bigram?
$$
Trigram: p(s_t\mid s_{t-2}, s_{t-1})
$$...
0
votes
0
answers
134
views
How to tell if the occurence of a variable is statistically significant/meaningful?
Since my previous question seems to be unclear, I will try to put it in other words.
I have been working on a child speech corpus, which consists of sentences in which children of various ages ...
- 82.7k
3
votes
1
answer
187
views
Breaking substitution cipher with language model
Frequency analysis is a common tool used to break substitution ciphers, but often relies on intuition and guesswork of a human. Since language models can objectively calculate perplexity (how ...
- 9,001
2
votes
1
answer
189
views
How to compute context-independent token representations in a biLM?
I've been reading this paper on ELMo word representations.
For context, here's my understanding of the standard bi-directional language model (biLM) thus far:
Given a sequence of tokens $(t_{1}, ...
CommunityBot
- 1
5
votes
2
answers
1k
views
Generating text from language model
I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be:
Apply softmax function
Take a weighted random choice to determine next word
This ...
- 1,923
5
votes
2
answers
12k
views
Understanding Add-1/Laplace smoothing with bigrams
I am working through an example of Add-1 smoothing in the context of NLP
Say that there is the following corpus (start and end tokens included)
...
CommunityBot
- 1
0
votes
0
answers
22
views
Choosing a model for input: categorised, weighted sequence, output: binary variable
What would be an appropriate model for predicting a binary target variable, given a weighted sequence?
Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
- 101
2
votes
1
answer
515
views
What's the best practice for dealing with OOV characters?
I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
- 82.7k
2
votes
1
answer
830
views
BERT masking - why does it require sampling, and how does it mitigate the mismatch of the [MASK] token when fine-tuning?
I'm reading the BERT paper and jalammar's illustrative guide for BERT.
I don't understand 2 things about the method's crux - the masked language model:
why does masking requires us to sample (take ...
- 6,912