All Questions
Tagged with language-models natural-language
77 questions
2
votes
1
answer
37
views
Probability Distribution Underlying Ngram Model
Texts introducing ngram models often directly manipulate conditional probabilities. For example, given a corpus $V$ with a bigram model on its words, we would compute the probability of a sentence $...
- 73
0
votes
0
answers
10
views
Ngram - Good Smoothing Probability Problem
I am working on creating an n-gram model for unigrams, bigrams, and trigrams. However, I have two questions:
Should the sum of the unseen probability and the observed probability always equal 1?
If 𝑁�...
- 1
0
votes
0
answers
296
views
Bert Used for generative AI
I have a doubt regarding using "Bert" as a generative model. I know Bert can be used for classification or fine-tuning the question-answering. However, is it possible to use Bert to generate ...
- 175
0
votes
0
answers
47
views
Classification in BERT - why not use class as a feature?
I am currently following this post, which details how BERT was trained. I had a few questions about the classification task:
In the post, it mentions that the authors of BERT decided to add ...
- 319
1
vote
0
answers
91
views
Why does the best performing adapter-based parameter-efficient fine-tuning depend on the language model being fine-tuned?
https://arxiv.org/abs/2304.01933 shows that the best performing adapter-based parameter-efficient fine-tuning depends on the language model being fine-tuned:
E.g., LORA is the best adapter for LlaMa-...
- 47.6k
1
vote
1
answer
94
views
Intuitive difference between NN and attention for text prediction
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:
Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
0
votes
1
answer
14
views
Continuous Bag of Words derivation
The continuous bag of words model has the following log probability for observing a sequence of words: $$\log P(\textbf{w})=\sum_{c=1}^{C}\log{P(w_c|w_{c-m},...w_{c-1}, w_{c+1},...,w_{c+m}})$$
I don't ...
- 319
0
votes
1
answer
34
views
Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)
I understand how later neural language models (such as those used in the Word2Vec papers) framed the language modelling problem in a self-supervised way by learning to predict the next word (or any ...
- 1,046
0
votes
1
answer
2k
views
Difference between "transfer learning" and "fine-tuning"
I am currently reading the BERT paper and it splits the use of pre-trained models into two categories:
Feature-based whereby you just take the embeddings for the tokens and plug it into whatever ...
- 1,046
0
votes
1
answer
484
views
Perplexity for different n-gram models
I'm training a Lidstone Model with different n-gram sequences to see witch one is the best (2-gram, 3-gram, 4-gram, etc) in the same text database.
When I give all these models an unseen text sample ...
- 3
2
votes
1
answer
403
views
Do I need training data in multiple languages for a multilingual transformer?
I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.
...
- 465
1
vote
1
answer
468
views
How to handle out of the bag token in NLP?
In my current language model my model is unaware of any token that is out-of-bag
for example:- In my summary generating model when we pass some token that is out-of-bag then my model will completely ...
- 13
2
votes
1
answer
46
views
Language model gives different results when Bayes' theorem is applied
Say, the following example is our corpus:
a quick brown fox jumps over the lazy dog.
Here, there are 9 tokens in the corpus. ...
- 235
1
vote
3
answers
438
views
How to improve language model ex: BERT on unseen text in training?
so I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure.
However, when I remove a ...
- 11
2
votes
0
answers
622
views
Variable batch size for inputs of different length
We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
2
votes
1
answer
349
views
Does the transformers model (in “Attention is All You Need”) exclude the encoder in language modelling tasks?
The language model I am referring to is the one outlined in "Attention is All You Need":
My understanding is (please correct me if I am wrong) that when the task is translation, the encoder'...
- 23
2
votes
2
answers
1k
views
Calculating perplexity with smoothing techniques (NLP)
This question is about smoothed n-gram language models. When we use additive smoothing on the train set to determine the conditional probabilities, and calculate the perplexity of train data, where ...
- 23
8
votes
1
answer
573
views
Why are Transformers "suboptimal" for language modeling but not for translation?
Language Models with Transformers states:
Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to ...
- 1,347
2
votes
1
answer
982
views
For BERT-based models, do we absolutely have to include the [CLS] and [SEP] special tokens in the input data?
The thought just occurred to me while I was processing data. If we're using the [CLS] token for classification, then it would obviously make sense to include it, ...
- 4,177
7
votes
2
answers
2k
views
Why are language modeling pre-training objectives considered unsupervised?
Maybe this is stemming from my not-so-great grasp of supervised vs. unsupervised learning, but my understanding is that if we have access to ground-truth labels then it's supervised learning and if ...
- 4,177
3
votes
1
answer
187
views
Breaking substitution cipher with language model
Frequency analysis is a common tool used to break substitution ciphers, but often relies on intuition and guesswork of a human. Since language models can objectively calculate perplexity (how ...
3
votes
1
answer
4k
views
Use of ignore_index on CrossEntropyLoss() for text models
I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this.
From what I understand ...
- 33
1
vote
0
answers
38
views
Language Model dealing with Book Dialogue
My goal is to generate text based on a specific book using an LSTM language model. One problem with my generator is that it seems that the book's dialogue is somewhat messing up my generator. I had ...
5
votes
2
answers
1k
views
Generating text from language model
I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be:
Apply softmax function
Take a weighted random choice to determine next word
This ...
2
votes
1
answer
515
views
What's the best practice for dealing with OOV characters?
I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
- 33
1
vote
0
answers
2k
views
Which dimensionality reduction technique works well for BERT sentence embeddings?
I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
- 11
3
votes
0
answers
1k
views
Different ways to calculate pointwise mutual information for word co-occurrence [closed]
I have a (very) small corpus of documents. As a representative example: 450 documents, 280000 total word count.
I am calculating Positive Pointwise Mutual Information (PPMI) between a selection of ...
- 133
2
votes
1
answer
830
views
BERT masking - why does it require sampling, and how does it mitigate the mismatch of the [MASK] token when fine-tuning?
I'm reading the BERT paper and jalammar's illustrative guide for BERT.
I don't understand 2 things about the method's crux - the masked language model:
why does masking requires us to sample (take ...
- 3,360
2
votes
3
answers
790
views
Confusion about CBOW and Skip-Gram models?
I've read a couple online description of CBOW and Skip-Gram and usually the descriptions starts like this:
We need to train models on words
So we encode words using vectors
One-hot encoding is not ...
- 1,630
1
vote
0
answers
1k
views
Inference time for text genration using fine-tuned gpt2
I have re-trained GPT2 model using over 10 million sentences for QA. And while testing also I am getting very good results. The only problem now I am facing is that I have millions of test data that I ...
- 121
1
vote
0
answers
201
views
Language Identification Better Results with Unigrams
I have a school project which consists of identifying each language of a tweet from a dataset of tweets. The dataset contains tweets in Spanish, Portuguese, English, Basque, Galician and Catalan. The ...
user255085
1
vote
1
answer
312
views
Skip-gram model multiplicative constant in the objective function?
I was reading this paper (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) I cannot understand where does the multiplicative constant $...
- 271
0
votes
1
answer
80
views
Can a deep learning model learn to understand/interpret the logical sentences?
I don't know what the development in NLP has gone so far. I'm curious about whether my machine can answer me following question?
"Today is Sunday. I'll finish my project in two days. What will be day ...
- 155
0
votes
1
answer
86
views
get predictability of word given sentence in python
I have a paragraph and I want to get the probability (p(word | context) ) of each word, given previous words, for various models (e.g. pre-trained LSTM).
Where can pretrained models would allow me to ...
- 57
1
vote
1
answer
86
views
Do recurrent neural language models greedily model language probability?
Want to check my understanding of recurrent neural language models (in this case I'm working with a decoder in an encoder-decoder RNN but I don't think that matters significantly). I'm trying to ...
- 13
1
vote
0
answers
31
views
Contextual word embeddings to estimate likelihood of word given previous words in sentence?
I'd like to use contextual embeddings to estimate the likelihood of word n given the previous n-1 words in a sentence. Which pretrained models would allow me to do this (could I use something like ...
- 11
5
votes
1
answer
1k
views
Denoising autoencoders vs masked approaches
I am not an expert in language modelling domain.
There are mainly two approaches that are being used nowadays. Denoising autoencoders and ...
- 378
0
votes
1
answer
220
views
How to train a RNN language model?
I want to train a RNN-based language model from https://arxiv.org/pdf/1409.2329.pdf for next word prediction.
How to split the sentences from the dataset into input and ground truth during the ...
- 1
0
votes
1
answer
62
views
Difference between ELMo and a normal deep network
I was reading up about ELMo and what I could gather was that we essentially combine the weights from different stacked lstm/gru layers for a given token as different layers are suited for different ...
- 101
0
votes
1
answer
195
views
Likelihood for a test data (sequence of characters) given two unigram models
I would like to find the likelihood of a sequence of characters (the test data), given two unigram models.
The sequence (test data) is:
A B C B B
The models ...
- 101
0
votes
1
answer
27
views
Modeling words in a language based on their characters
I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
- 13
0
votes
1
answer
112
views
CMUDict normalized for word frequency
I am trying to train a neural grapheme to phoneme (G2P) model on CMUDict, but I find that pretty soon its loss is barely decreasing. Also, when I train the model on a different similar-sized dataset (...
- 101
2
votes
1
answer
730
views
How to sample a language model?
I've successfully trained a language model using LSTMs. But I have a confusion about sampling.
On sampling, we generate a probability distribution at each time step. It will be of length vocabulary ...
- 123
1
vote
1
answer
384
views
Where can I find pre-trained language models in English and German?
Where can I find (more) pre-trained language models? I am especially interested in neural network based models for English and German.
I am aware only of Language Model on One Billion Word Benchmark ...
1
vote
1
answer
206
views
How does clustering improve a language model?
This article describes a hierarchical clustering algorithm which clusters the words within a vocabulary based on their similarity, in order to improve a language model (in the article, n-grams).
How ...
- 359
3
votes
0
answers
39
views
natural language understanding algorithms [closed]
I am doing some research into how smart personal assistants work like siri, alexa etc. I have found that it is using automatic speech recognition to turn the speech into weighted text form and then ...
- 31
2
votes
2
answers
120
views
What are commonly used methods to represent a document by a vector?
Methods that I know of
Bag of words + weighting: tf-idf, bm25
Topic models: LSA, LDA
Word/sentence/document embedding
Are there other commonly used methods to represent a document by a vector?
- 16.8k
2
votes
1
answer
189
views
How to compute context-independent token representations in a biLM?
I've been reading this paper on ELMo word representations.
For context, here's my understanding of the standard bi-directional language model (biLM) thus far:
Given a sequence of tokens $(t_{1}, ...
1
vote
1
answer
42
views
What the 'N' means on the following N-gram approximation
What the 'N'(capital) means in the following N-gram
approximation to the conditional probability of the next word in a sequence ?
- 111
1
vote
0
answers
35
views
Character level RNN for converting word forms
I want to build a char RNN to convert word form from one to another, for example, singular nouns such as lion to lions. However ...