Questions tagged [language-models]
A statistical language model is a probability distribution over sequences of words.
34 questions with no upvoted or accepted answers
3
votes
1
answer
220
views
How do we pass data to a RNN?
Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we:
Scheme 1
Take the first batch of ...
- 2,683
2
votes
2
answers
44
views
End-Tokens are Required to make Ngram Models Proper
The standard bigram model, (for example defined here) defines a probability distribution over a corpus $V$ based on the following principles:
The marginal probability of a word $w$ is defined as its ...
- 73
2
votes
0
answers
622
views
Variable batch size for inputs of different length
We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
2
votes
0
answers
451
views
Deciding on the training sequences for RNN/LSTM language model
In a character language model, text is seen as a stream of characters. Say we have a training text as a string s, with length ...
- 692
2
votes
0
answers
2k
views
Computing Training Set Perplexity of a Neural Language Model: Too low values
I am implementing a Language Model based on a Deep Learning architecture (RNN+Softmax).
The cost function I am using is the cross-entropy between the vector of probabilities at the softmax layer and ...
- 71
2
votes
0
answers
272
views
Alpha on Katz Backoff using Simple Good-Turing
I'm building an n-gram language model to predict the next word, I've implemented a simple Good-Turing smoothing on all my probabilities and have calculated the P0(mass probability of unseen event).
I ...
- 41
1
vote
0
answers
91
views
Why does the best performing adapter-based parameter-efficient fine-tuning depend on the language model being fine-tuned?
https://arxiv.org/abs/2304.01933 shows that the best performing adapter-based parameter-efficient fine-tuning depends on the language model being fine-tuned:
E.g., LORA is the best adapter for LlaMa-...
- 47.6k
1
vote
0
answers
91
views
What's the role of masking in transformer and BERT?
I've recently implemented the architectures of Transformer and BERT and found that they both have common property - masked layers among one of them. I have come to questions like below.
As far as I ...
- 123
1
vote
0
answers
38
views
Language Model dealing with Book Dialogue
My goal is to generate text based on a specific book using an LSTM language model. One problem with my generator is that it seems that the book's dialogue is somewhat messing up my generator. I had ...
1
vote
0
answers
2k
views
Which dimensionality reduction technique works well for BERT sentence embeddings?
I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
- 11
1
vote
0
answers
1k
views
Inference time for text genration using fine-tuned gpt2
I have re-trained GPT2 model using over 10 million sentences for QA. And while testing also I am getting very good results. The only problem now I am facing is that I have millions of test data that I ...
- 121
1
vote
0
answers
201
views
Language Identification Better Results with Unigrams
I have a school project which consists of identifying each language of a tweet from a dataset of tweets. The dataset contains tweets in Spanish, Portuguese, English, Basque, Galician and Catalan. The ...
user255085
1
vote
0
answers
31
views
Contextual word embeddings to estimate likelihood of word given previous words in sentence?
I'd like to use contextual embeddings to estimate the likelihood of word n given the previous n-1 words in a sentence. Which pretrained models would allow me to do this (could I use something like ...
- 11
1
vote
0
answers
516
views
Perplexity for short sentences
I have a model that outputs short sentences and want to compare the quality of its outputs for different configurations by computing their perplexities using another model.
I tried to use the gpt-2 ...
- 11
1
vote
0
answers
35
views
Character level RNN for converting word forms
I want to build a char RNN to convert word form from one to another, for example, singular nouns such as lion to lions. However ...
1
vote
0
answers
411
views
Feature extraction from strings
We are given a big collection of strings, and an intensity associated with each string in the collection. In a sense, a 'distribution' on the dictionary. We are given that the intensity of each string ...
1
vote
1
answer
114
views
Under periodic BPTT, is softmax evaluated only at the end of the period?
Suppose I have a continuous sequence $X$ of words and I wish to train a RNN language model. According to [1], I would split $X$ into subsequences $X^{1..|X|/k_1}$ $k_1$ sized subsequences ($k_1$ is ...
- 210
0
votes
0
answers
10
views
Ngram - Good Smoothing Probability Problem
I am working on creating an n-gram model for unigrams, bigrams, and trigrams. However, I have two questions:
Should the sum of the unseen probability and the observed probability always equal 1?
If 𝑁�...
- 1
0
votes
0
answers
12
views
Validation accuracy dip and recovery when restarting training
i was fine-tuning this large language model with Stochastic Gradient Descent and mid epoch i stopped training, and saved the model weights. Then at a later time, reloaded the weights and restarted the ...
- 348
0
votes
0
answers
248
views
Is it necessary to use attention mask for mean pooling for BERT?
I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
- 138
0
votes
0
answers
20
views
Why doesn't BERT give me back my original sentence?
I've started playing with BERT encoder through the huggingface module.
I passed it a normal unmasked sentence and got the ...
- 641
0
votes
0
answers
296
views
Bert Used for generative AI
I have a doubt regarding using "Bert" as a generative model. I know Bert can be used for classification or fine-tuning the question-answering. However, is it possible to use Bert to generate ...
- 175
0
votes
0
answers
47
views
Classification in BERT - why not use class as a feature?
I am currently following this post, which details how BERT was trained. I had a few questions about the classification task:
In the post, it mentions that the authors of BERT decided to add ...
- 319
0
votes
1
answer
14
views
Continuous Bag of Words derivation
The continuous bag of words model has the following log probability for observing a sequence of words: $$\log P(\textbf{w})=\sum_{c=1}^{C}\log{P(w_c|w_{c-m},...w_{c-1}, w_{c+1},...,w_{c+m}})$$
I don't ...
- 319
0
votes
0
answers
22
views
Choosing a model for input: categorised, weighted sequence, output: binary variable
What would be an appropriate model for predicting a binary target variable, given a weighted sequence?
Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
- 101
0
votes
1
answer
3k
views
Finding the perplexity of multiple examples
I am trying to find a way to calculate perplexity of a language model of multiple 3-word examples from my test set, or perplexity of the corpus of the test set. As the test set, I have a paragraph ...
0
votes
1
answer
86
views
get predictability of word given sentence in python
I have a paragraph and I want to get the probability (p(word | context) ) of each word, given previous words, for various models (e.g. pre-trained LSTM).
Where can pretrained models would allow me to ...
- 57
0
votes
0
answers
13
views
How can an untrusted feature value affect the accuracy of a machine learning model?
I'm building a language detector model to classify between 2 languages given text image. The input will be the image, but I have another feature that can tell somehow the language (between 0,1) ...
- 101
0
votes
1
answer
27
views
Modeling words in a language based on their characters
I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
- 13
0
votes
1
answer
112
views
CMUDict normalized for word frequency
I am trying to train a neural grapheme to phoneme (G2P) model on CMUDict, but I find that pretty soon its loss is barely decreasing. Also, when I train the model on a different similar-sized dataset (...
- 101
0
votes
0
answers
98
views
Normalize probability distribution by variance of each class
I have several topics that I will collect language data for. Using Mturk I will ask responders to write sentences for each topic. The sentences will be used to train a language model. Language models ...
- 51
0
votes
0
answers
75
views
Decide threshold for LM to filter out ill-formed sentence
I have a corpus of sentences. This corpus contains well-formed as well as ill-formed sentences. I want to filter out ill-formed sentences from the corpus.
Ill-formed sentences are of two types :
...
0
votes
0
answers
51
views
Classifying Sentences when outcome is a sentence
I am completely new to text classification and facing the following problem, I want to classify sentences and have a labeled dataset, the thing is that my Y variable is itself a sentence, hence I ...
- 103
0
votes
0
answers
134
views
How to tell if the occurence of a variable is statistically significant/meaningful?
Since my previous question seems to be unclear, I will try to put it in other words.
I have been working on a child speech corpus, which consists of sentences in which children of various ages ...
- 101