Questions tagged [language-models]

A statistical language model is a probability distribution over sequences of words.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

How truncation works when applying BERT tokenizer on the batch of sentence pairs in HuggingFace? [closed]

Say, I have three sample sentences: ...
user avatar
0 votes
0 answers
21 views

Mini batches and loss in recurrent neural networks (RNNs)

Suppose that we have a sequence $\left\{x^{(k)}\right\}_{k = 1}^{N}$ and that we wish to use a RNN to predict the next element of the sequence given the previous elements of the sequence (e.g., a ...
user avatar
  • 101
0 votes
1 answer
37 views

Feedforward Neural Net Language Model - computational complexity (word2vec)

While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM): It consists of input, projection, hidden and output layers. At the ...
user avatar
0 votes
0 answers
9 views

Hybrid approach for text categorization (Rule based + ML)

I want to build a multi-label text categorization system of paper abstracts with about 20 categories. For many of the categories keyword based logical rules exist with a low false positive and medium ...
user avatar
  • 1,119
1 vote
1 answer
19 views

How to handle out of the bag token in NLP?

In my current language model my model is unaware of any token that is out-of-bag for example:- In my summary generating model when we pass some token that is out-of-bag then my model will completely ...
user avatar
0 votes
0 answers
15 views

What's the role of masking in transformer and BERT?

I've recently implemented the architectures of Transformer and BERT and found that they both have common property - masked layers among one of them. I have come to questions like below. As far as I ...
user avatar
  • 111
0 votes
0 answers
15 views

Pretrained Language model comparison with binary sentiment classification

On two independent datasets, I am comparing XLNet and BERT models with binary sentiment classification tasks: the Twitter dataset, where sentences are short, and the IMDB review dataset, where ...
user avatar
1 vote
1 answer
30 views

Language model gives different results when Bayes' theorem is applied

Say, the following example is our corpus: a quick brown fox jumps over the lazy dog. Here, there are 9 tokens in the corpus. ...
user avatar
  • 185
0 votes
0 answers
11 views

Active logits length differ after augmentation in mean teacher setup for semi supervised learning

I am planning to apply mean-teacher for my problem of token classification. Since adding different noise for teacher and student is really important for the approach, i am confused about how to ...
user avatar
0 votes
0 answers
6 views

Methods for selecting best data points for training language models

I have a neural network model (a language model) that is already pretrained using a huge amount of data (think of BERT, for example.) I added some domain-related data (~120,000 extra text sentences) ...
user avatar
  • 133
0 votes
0 answers
13 views

Calculating RNN loss (for a SINGLE example) as a sum of individual time step losses VS. an average of individual time step losses [duplicate]

In Andrew Ng's course, I see RNN loss being calculated as a sum of the losses from each time step as seen here: In Stanford's CS224N, I see loss calculated as an average of individual losses as seen ...
user avatar
  • 1
1 vote
3 answers
60 views

How to improve language model ex: BERT on unseen text in training?

so I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure. However, when I remove a ...
user avatar
1 vote
0 answers
84 views

Variable batch size for inputs of different length

We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
user avatar
0 votes
0 answers
9 views

Masked language models: can you train on remasked data?

Masked language models like BERT and friends are trained on the task of predicting words removed from input text. Normally, this text is removed at random from some training data. As far as I can tell ...
user avatar
  • 131
0 votes
0 answers
31 views

Issue related to Transformer decoder druing Inference using all previous output tokens until each decoder time step

I've been trying to understand the shapes used during decoder (both self-attention and enc-dec-attention blocks) and understand there is a difference in the way decoder runs during training versus ...
user avatar
  • 309
0 votes
0 answers
39 views

Three-way log likelihood ratio (LLR) / G-test

I have word counts and trigram counts for a text dataset. I want to calculate co-occurences, i.e. "best" trigrams, that most probably represent multiword expressions, using log likelihood ...
user avatar
  • 179
0 votes
0 answers
10 views

Understanding the neural network in the paper "words can shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors"

In this interesting paper, they show say that they have trained several LSTMs and then extracted the last layer features for their next steps. When training a neural network, we need to have a goal. ...
user avatar
  • 365
0 votes
0 answers
16 views

Single-value alpha parameter for Dirichlet distribution

I'm trying to implement an event schema induction method from a paper from 2015. The authors use a generative approach to learn a language model. For this, they use a lot of probability distributions ...
user avatar
  • 101
0 votes
0 answers
5 views

Intrinsic metrics to evaluate pretrained language models?

I learned that there are intrinsic and extrinsic evaluation methods for vector models. Although the most important evaluation is the extrinsic the intrinsic metrics are also useful. There are three ...
user avatar
  • 5,181
1 vote
1 answer
51 views

multiple likely ys for one instance of x: word prediction with LSTM

I have a ML project that is about predicting (suggesting) the next word based on the last n words, using LSTM. The output is a softmax dense layer the size of the vocabulary that shows the probability ...
user avatar
4 votes
1 answer
199 views

How does one design a custom loss function? What features make a loss function "good"?

I have a custom situation for which I am trying to design a cost function. The idea is that you have a stack of LSTMs doing something slightly unconventional. Each LSTM$_l$ computes a linear ...
user avatar
  • 153
2 votes
1 answer
124 views

Does the transformers model (in “Attention is All You Need”) exclude the encoder in language modelling tasks?

The language model I am referring to is the one outlined in "Attention is All You Need": My understanding is (please correct me if I am wrong) that when the task is translation, the encoder'...
user avatar
2 votes
1 answer
254 views

Calculating perplexity with smoothing techniques (NLP)

This question is about smoothed n-gram language models. When we use additive smoothing on the train set to determine the conditional probabilities, and calculate the perplexity of train data, where ...
user avatar
6 votes
1 answer
219 views

Why are Transformers "suboptimal" for language modeling but not for translation?

Language Models with Transformers states: Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to ...
user avatar
  • 1,173
1 vote
1 answer
275 views

For BERT-based models, do we absolutely have to include the [CLS] and [SEP] special tokens in the input data?

The thought just occurred to me while I was processing data. If we're using the [CLS] token for classification, then it would obviously make sense to include it, ...
user avatar
  • 2,371
3 votes
1 answer
623 views

Why are language modeling pre-training objectives considered unsupervised?

Maybe this is stemming from my not-so-great grasp of supervised vs. unsupervised learning, but my understanding is that if we have access to ground-truth labels then it's supervised learning and if ...
user avatar
  • 2,371
3 votes
1 answer
49 views

Breaking substitution cipher with language model

Frequency analysis is a common tool used to break substitution ciphers, but often relies on intuition and guesswork of a human. Since language models can objectively calculate perplexity (how ...
user avatar
2 votes
1 answer
1k views

Use of ignore_index on CrossEntropyLoss() for text models

I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this. From what I understand ...
user avatar
  • 23
4 votes
2 answers
492 views

Generating text from language model

I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be: Apply softmax function Take a weighted random choice to determine next word This ...
user avatar
0 votes
0 answers
19 views

Choosing a model for input: categorised, weighted sequence, output: binary variable

What would be an appropriate model for predicting a binary target variable, given a weighted sequence? Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
user avatar
  • 101
1 vote
1 answer
151 views

What's the best practice for dealing with OOV characters?

I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
user avatar
0 votes
1 answer
996 views

Finding the perplexity of multiple examples

I am trying to find a way to calculate perplexity of a language model of multiple 3-word examples from my test set, or perplexity of the corpus of the test set. As the test set, I have a paragraph ...
user avatar
1 vote
0 answers
1k views

Which dimensionality reduction technique works well for BERT sentence embeddings?

I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
user avatar
2 votes
0 answers
633 views

Different ways to calculate pointwise mutual information for word co-occurrence

I have a (very) small corpus of documents. As a representative example: 450 documents, 280000 total word count. I am calculating Positive Pointwise Mutual Information (PPMI) between a selection of ...
user avatar
  • 103
2 votes
1 answer
310 views

BERT masking - why does it require sampling, and how does it mitigate the mismatch of the [MASK] token when fine-tuning?

I'm reading the BERT paper and jalammar's illustrative guide for BERT. I don't understand 2 things about the method's crux - the masked language model: why does masking requires us to sample (take ...
user avatar
  • 2,606
1 vote
3 answers
318 views

Confusion about CBOW and Skip-Gram models?

I've read a couple online description of CBOW and Skip-Gram and usually the descriptions starts like this: We need to train models on words So we encode words using vectors One-hot encoding is not ...
user avatar
  • 1,071
1 vote
0 answers
710 views

Inference time for text genration using fine-tuned gpt2

I have re-trained GPT2 model using over 10 million sentences for QA. And while testing also I am getting very good results. The only problem now I am facing is that I have millions of test data that I ...
user avatar
1 vote
0 answers
96 views

Language Identification Better Results with Unigrams

I have a school project which consists of identifying each language of a tweet from a dataset of tweets. The dataset contains tweets in Spanish, Portuguese, English, Basque, Galician and Catalan. The ...
user avatar
1 vote
1 answer
116 views

Skip-gram model multiplicative constant in the objective function?

I was reading this paper (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) I cannot understand where does the multiplicative constant $...
user avatar
  • 261
1 vote
1 answer
675 views

Cross layer parameter sharing in ALBERT Model

I am reading the paper "ALBERT: LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS". ALBERT uses cross-layer parameter sharing to improve model performance. I don't ...
user avatar
  • 163
2 votes
1 answer
48 views

Machine Translation: with sufficient parallel data, can we improve even further the performance of the system with the use of monolingual data?

I am trying to find scientific literature that studies if, in a situation in which we already have enough parallel data, the addition of monolingual data can further improve performance. I have not ...
user avatar
0 votes
1 answer
46 views

Can a deep learning model learn to understand/interpret the logical sentences?

I don't know what the development in NLP has gone so far. I'm curious about whether my machine can answer me following question? "Today is Sunday. I'll finish my project in two days. What will be day ...
user avatar
0 votes
1 answer
57 views

get predictability of word given sentence in python

I have a paragraph and I want to get the probability (p(word | context) ) of each word, given previous words, for various models (e.g. pre-trained LSTM). Where can pretrained models would allow me to ...
user avatar
  • 27
1 vote
1 answer
60 views

Do recurrent neural language models greedily model language probability?

Want to check my understanding of recurrent neural language models (in this case I'm working with a decoder in an encoder-decoder RNN but I don't think that matters significantly). I'm trying to ...
user avatar
5 votes
1 answer
1k views

Why can't standard conditional language models be trained left-to-right *and* right-to-left?

From the BERT paper: Unfortunately, standard conditional language models can only be trained left-to-right or right-to-left, since bidirectional conditioning would allow each word to indirectly “...
user avatar
  • 1,226
0 votes
0 answers
12 views

How can an untrusted feature value affect the accuracy of a machine learning model?

I'm building a language detector model to classify between 2 languages given text image. The input will be the image, but I have another feature that can tell somehow the language (between 0,1) ...
user avatar
1 vote
0 answers
29 views

Contextual word embeddings to estimate likelihood of word given previous words in sentence?

I'd like to use contextual embeddings to estimate the likelihood of word n given the previous n-1 words in a sentence. Which pretrained models would allow me to do this (could I use something like ...
user avatar
1 vote
0 answers
300 views

Perplexity for short sentences

I have a model that outputs short sentences and want to compare the quality of its outputs for different configurations by computing their perplexities using another model. I tried to use the gpt-2 ...
user avatar
  • 11
0 votes
1 answer
68 views

Neural language model: Derivation of MLE

Recently, I studied NNLM and I saw the derivation of softmax using MLE: \begin{align} & \frac{\partial\log P(w_t\mid h)}{\partial\theta} \\[8pt] = {} & \frac{\partial \log \exp(s_\theta(w_t,...
user avatar
3 votes
1 answer
501 views

Denoising autoencoders vs masked approaches

I am not an expert in language modelling domain. There are mainly two approaches that are being used nowadays. Denoising autoencoders and ...
user avatar
  • 163