Questions tagged [language-models]
A statistical language model is a probability distribution over sequences of words.
109
questions
0
votes
0
answers
6
views
Importance of language models when given simple instruction to a Robot
Do I really need a transformer-based language model if I want to instruct a robot very simple task?
Suppose I give an instruction like "Go to the right and pick up the book from the table".
...
- 175
0
votes
0
answers
40
views
Bert Used for generative AI
I have a doubt regarding using "Bert" as a generative model. I know Bert can be used for classification or fine-tuning the question-answering. However, is it possible to use Bert to generate ...
- 175
0
votes
0
answers
17
views
Classification in BERT - why not use class as a feature?
I am currently following this post, which details how BERT was trained. I had a few questions about the classification task:
In the post, it mentions that the authors of BERT decided to add ...
- 269
0
votes
0
answers
10
views
How does Kneser Ney estimate ngrams with BOS without dividing by zero?
The recursive formula for (unmodified) Kneser Ney smoothing is (per Jurafsky08 3.40)
$P_{KN}(w_i | h) = \frac{\text{max}(c_{KN}(h\ w_i) - d, 0)}{\sum_v c_{KN}(h \ v )} + \lambda(h) P_{KN}(w_i | h)$
...
- 1,366
0
votes
0
answers
21
views
Implementation of spBLEU
I was looking for a way to compute statistics for evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does ...
- 21
0
votes
0
answers
6
views
is there an existing language model for calculating the probability of P(output|input), needed for aligning texts from OCR
I have a bunch of boxes containing texts from receipt OCR results, and I need to a way to align them properly.
The attached photo shows part of a contrived receipt,
but it illustrates the problem I ...
- 1,115
4
votes
3
answers
742
views
How do you add knowledge to LLMs
I recently heard an interesting comment from a gentleman on YouTube and it made sense instantly.
To paraphrase he explained that "fine-tuning" an LM is not necessarily adding knowledge to a ...
- 643
1
vote
0
answers
68
views
Why does the best performing adapter-based parameter-efficient fine-tuning depend on the language model being fine-tuned?
https://arxiv.org/abs/2304.01933 shows that the best performing adapter-based parameter-efficient fine-tuning depends on the language model being fine-tuned:
E.g., LORA is the best adapter for LlaMa-...
- 45.9k
0
votes
0
answers
22
views
Huge overfit on prediction model -due to data with low predictive power or can this be fixed? (Python)
I am not used to working with machine learning models, and are currently sitting with an issue i hope you can help me with.
I am sitting with a multi classification problem, where i try to predict ...
- 1
0
votes
0
answers
347
views
What the context window size in pre-train gpt-2
I have questions about GPT-2.
What is the context window in the pre-train gpt-2 train?
As I know language modeling is based on gpt-2 use prefix language modeling that is demonstrated in picture form ...
- 1
0
votes
0
answers
148
views
Perplexity with log probabilities
I want to calculate the perplexity of my language model. To avoid underflow, I stored the log of the probabilities. Since the probabilities are between 0 and 1, the log of them is negative. So when I ...
- 111
1
vote
1
answer
68
views
Intuitive difference between NN and attention for text prediction
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:
Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
0
votes
1
answer
10
views
Continuous Bag of Words derivation
The continuous bag of words model has the following log probability for observing a sequence of words: $$\log P(\textbf{w})=\sum_{c=1}^{C}\log{P(w_c|w_{c-m},...w_{c-1}, w_{c+1},...,w_{c+m}})$$
I don't ...
- 269
0
votes
1
answer
26
views
Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)
I understand how later neural language models (such as those used in the Word2Vec papers) framed the language modelling problem in a self-supervised way by learning to predict the next word (or any ...
- 1,014
0
votes
1
answer
1k
views
Difference between "transfer learning" and "fine-tuning"
I am currently reading the BERT paper and it splits the use of pre-trained models into two categories:
Feature-based whereby you just take the embeddings for the tokens and plug it into whatever ...
- 1,014
5
votes
1
answer
902
views
Should a language model like GPT-3 be directly used to perform classification?
The OpenAI API enables classification by sampling from GPT-3 given a prompt. Is estimating posterior probabilities a more statistically sound approach? Below is a specification of what "...
- 1,260
0
votes
1
answer
240
views
Perplexity for different n-gram models
I'm training a Lidstone Model with different n-gram sequences to see witch one is the best (2-gram, 3-gram, 4-gram, etc) in the same text database.
When I give all these models an unseen text sample ...
- 3
1
vote
1
answer
194
views
Do I need training data in multiple languages for a multilingual transformer?
I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.
...
- 385
0
votes
1
answer
245
views
Feedforward Neural Net Language Model - computational complexity (word2vec)
While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM):
It consists of input, projection, hidden and output layers. At the ...
- 21
1
vote
1
answer
376
views
How to handle out of the bag token in NLP?
In my current language model my model is unaware of any token that is out-of-bag
for example:- In my summary generating model when we pass some token that is out-of-bag then my model will completely ...
- 13
1
vote
0
answers
79
views
What's the role of masking in transformer and BERT?
I've recently implemented the architectures of Transformer and BERT and found that they both have common property - masked layers among one of them. I have come to questions like below.
As far as I ...
- 123
2
votes
1
answer
38
views
Language model gives different results when Bayes' theorem is applied
Say, the following example is our corpus:
a quick brown fox jumps over the lazy dog.
Here, there are 9 tokens in the corpus. ...
- 215
1
vote
3
answers
362
views
How to improve language model ex: BERT on unseen text in training?
so I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure.
However, when I remove a ...
- 11
2
votes
0
answers
552
views
Variable batch size for inputs of different length
We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
2
votes
1
answer
194
views
multiple likely ys for one instance of x: word prediction with LSTM
I have a ML project that is about predicting (suggesting) the next word based on the last n words, using LSTM. The output is a softmax dense layer the size of the vocabulary that shows the probability ...
- 51
5
votes
1
answer
1k
views
How does one design a custom loss function? What features make a loss function "good"?
I have a custom situation for which I am trying to design a cost function.
The idea is that you have a stack of LSTMs doing something slightly unconventional. Each LSTM$_l$ computes a linear ...
- 245
2
votes
1
answer
309
views
Does the transformers model (in “Attention is All You Need”) exclude the encoder in language modelling tasks?
The language model I am referring to is the one outlined in "Attention is All You Need":
My understanding is (please correct me if I am wrong) that when the task is translation, the encoder'...
- 23
2
votes
2
answers
927
views
Calculating perplexity with smoothing techniques (NLP)
This question is about smoothed n-gram language models. When we use additive smoothing on the train set to determine the conditional probabilities, and calculate the perplexity of train data, where ...
- 23
7
votes
1
answer
451
views
Why are Transformers "suboptimal" for language modeling but not for translation?
Language Models with Transformers states:
Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to ...
- 1,273
2
votes
1
answer
770
views
For BERT-based models, do we absolutely have to include the [CLS] and [SEP] special tokens in the input data?
The thought just occurred to me while I was processing data. If we're using the [CLS] token for classification, then it would obviously make sense to include it, ...
- 3,466
7
votes
2
answers
2k
views
Why are language modeling pre-training objectives considered unsupervised?
Maybe this is stemming from my not-so-great grasp of supervised vs. unsupervised learning, but my understanding is that if we have access to ground-truth labels then it's supervised learning and if ...
- 3,466
3
votes
1
answer
160
views
Breaking substitution cipher with language model
Frequency analysis is a common tool used to break substitution ciphers, but often relies on intuition and guesswork of a human. Since language models can objectively calculate perplexity (how ...
3
votes
1
answer
3k
views
Use of ignore_index on CrossEntropyLoss() for text models
I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this.
From what I understand ...
- 33
1
vote
0
answers
32
views
Language Model dealing with Book Dialogue
My goal is to generate text based on a specific book using an LSTM language model. One problem with my generator is that it seems that the book's dialogue is somewhat messing up my generator. I had ...
5
votes
2
answers
1k
views
Generating text from language model
I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be:
Apply softmax function
Take a weighted random choice to determine next word
This ...
0
votes
0
answers
20
views
Choosing a model for input: categorised, weighted sequence, output: binary variable
What would be an appropriate model for predicting a binary target variable, given a weighted sequence?
Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
- 101
1
vote
1
answer
418
views
What's the best practice for dealing with OOV characters?
I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
- 23
0
votes
1
answer
3k
views
Finding the perplexity of multiple examples
I am trying to find a way to calculate perplexity of a language model of multiple 3-word examples from my test set, or perplexity of the corpus of the test set. As the test set, I have a paragraph ...
1
vote
0
answers
2k
views
Which dimensionality reduction technique works well for BERT sentence embeddings?
I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
- 11
3
votes
0
answers
1k
views
Different ways to calculate pointwise mutual information for word co-occurrence [closed]
I have a (very) small corpus of documents. As a representative example: 450 documents, 280000 total word count.
I am calculating Positive Pointwise Mutual Information (PPMI) between a selection of ...
- 133
2
votes
1
answer
711
views
BERT masking - why does it require sampling, and how does it mitigate the mismatch of the [MASK] token when fine-tuning?
I'm reading the BERT paper and jalammar's illustrative guide for BERT.
I don't understand 2 things about the method's crux - the masked language model:
why does masking requires us to sample (take ...
- 3,200
2
votes
3
answers
662
views
Confusion about CBOW and Skip-Gram models?
I've read a couple online description of CBOW and Skip-Gram and usually the descriptions starts like this:
We need to train models on words
So we encode words using vectors
One-hot encoding is not ...
- 1,440
1
vote
0
answers
1k
views
Inference time for text genration using fine-tuned gpt2
I have re-trained GPT2 model using over 10 million sentences for QA. And while testing also I am getting very good results. The only problem now I am facing is that I have millions of test data that I ...
- 121
1
vote
0
answers
196
views
Language Identification Better Results with Unigrams
I have a school project which consists of identifying each language of a tweet from a dataset of tweets. The dataset contains tweets in Spanish, Portuguese, English, Basque, Galician and Catalan. The ...
user255085
1
vote
1
answer
282
views
Skip-gram model multiplicative constant in the objective function?
I was reading this paper (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) I cannot understand where does the multiplicative constant $...
- 271
1
vote
1
answer
1k
views
Cross layer parameter sharing in ALBERT Model
I am reading the paper "ALBERT: LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS". ALBERT uses cross-layer parameter sharing to improve model performance. I don't ...
- 163
2
votes
1
answer
68
views
Machine Translation: with sufficient parallel data, can we improve even further the performance of the system with the use of monolingual data?
I am trying to find scientific literature that studies if, in a situation in which we already have enough parallel data, the addition of monolingual data can further improve performance.
I have not ...
- 23
0
votes
1
answer
79
views
Can a deep learning model learn to understand/interpret the logical sentences?
I don't know what the development in NLP has gone so far. I'm curious about whether my machine can answer me following question?
"Today is Sunday. I'll finish my project in two days. What will be day ...
- 155
0
votes
1
answer
82
views
get predictability of word given sentence in python
I have a paragraph and I want to get the probability (p(word | context) ) of each word, given previous words, for various models (e.g. pre-trained LSTM).
Where can pretrained models would allow me to ...
- 57
1
vote
1
answer
82
views
Do recurrent neural language models greedily model language probability?
Want to check my understanding of recurrent neural language models (in this case I'm working with a decoder in an encoder-decoder RNN but I don't think that matters significantly). I'm trying to ...
- 13