Questions tagged [language-models]

A statistical language model is a probability distribution over sequences of words.

Filter by
Sorted by
Tagged with
0
votes
1answer
15 views

For BERT-based models, do we absolutely have to include the [CLS] and [SEP] special tokens in the input data?

The thought just occurred to me while I was processing data. If we're using the [CLS] token for classification, then it would obviously make sense to include it, ...
3
votes
1answer
51 views

Why are language modeling pre-training objectives considered unsupervised?

Maybe this is stemming from my not-so-great grasp of supervised vs. unsupervised learning, but my understanding is that if we have access to ground-truth labels then it's supervised learning and if ...
2
votes
1answer
23 views

Breaking substitution cipher with language model

Frequency analysis is a common tool used to break substitution ciphers, but often relies on intuition and guesswork of a human. Since language models can objectively calculate perplexity (how ...
1
vote
1answer
90 views

Use of ignore_index on CrossEntropyLoss() for text models

I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this. From what I understand ...
4
votes
2answers
115 views

Generating text from language model

I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be: Apply softmax function Take a weighted random choice to determine next word This ...
0
votes
0answers
17 views

Choosing a model for input: categorised, weighted sequence, output: binary variable

What would be an appropriate model for predicting a binary target variable, given a weighted sequence? Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
1
vote
1answer
37 views

What's the best practice for dealing with OOV characters?

I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
0
votes
1answer
160 views

Finding the perplexity of multiple examples

I am trying to find a way to calculate perplexity of a language model of multiple 3-word examples from my test set, or perplexity of the corpus of the test set. As the test set, I have a paragraph ...
0
votes
0answers
25 views

Is there a way to provide multiple masks to BERT in MLM task?

I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence. ...
0
votes
0answers
27 views

Extract Keyword/Concept From Column Description Using NLP

Suppose in my database, each table has a description associated with each column and I want to further extract keyword or key concept from the description. For example, mean of transaction amount in ...
0
votes
0answers
63 views

Which dimensionality reduction technique works well for BERT sentence embeddings?

I'm trying to cluster hundreds of text documents so that each each cluster represents a distinct topic. Instead of using topic modeling (which I know I could do too), I want to follow a two-step ...
0
votes
0answers
20 views

Code2vec evaluation metrics: precision, recall, and F1 score over sub-tokens, case-insensitive

In the paper code2vec (https://arxiv.org/pdf/1803.09473.pdf) they use as evaluation metrics: precision, recall and f1 over subtokens. The authors say textually: "This is based on the idea that ...
2
votes
0answers
184 views

Different ways to calculate pointwise mutual information for word co-occurrence

I have a (very) small corpus of documents. As a representative example: 450 documents, 280000 total word count. I am calculating Positive Pointwise Mutual Information (PPMI) between a selection of ...
0
votes
0answers
30 views

Pre-requisites for GPT-3

What papers should I read before attempting to understand the GPT-3 paper (apart from the GPT-1 and GPT-2 papers)? Given that GPT-3 is a language model, I'm looking at NLP-related papers specifically. ...
1
vote
1answer
95 views

BERT masking - why does it require sampling, and how does it mitigate the mismatch of the [MASK] token when fine-tuning?

I'm reading the BERT paper and jalammar's illustrative guide for BERT. I don't understand 2 things about the method's crux - the masked language model: why does masking requires us to sample (take ...
0
votes
3answers
139 views

Confusion about CBOW and Skip-Gram models?

I've read a couple online description of CBOW and Skip-Gram and usually the descriptions starts like this: We need to train models on words So we encode words using vectors One-hot encoding is not ...
1
vote
0answers
328 views

Inference time for text genration using fine-tuned gpt2

I have re-trained GPT2 model using over 10 million sentences for QA. And while testing also I am getting very good results. The only problem now I am facing is that I have millions of test data that I ...
0
votes
0answers
112 views

Using perplexity to evaluate n gram model

I have built a simple n gram model for predicting the next word in a sentenced based on a maximum likelihood estimation. Currently I am using the Katz backoff algorithm to decide between n gram ...
1
vote
0answers
29 views

Language Identification Better Results with Unigrams

I have a school project which consists of identifying each language of a tweet from a dataset of tweets. The dataset contains tweets in Spanish, Portuguese, English, Basque, Galician and Catalan. The ...
1
vote
1answer
69 views

Skip-gram model multiplicative constant in the objective function?

I was reading this paper (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) I cannot understand where does the multiplicative constant $...
1
vote
1answer
406 views

Cross layer parameter sharing in ALBERT Model

I am reading the paper "ALBERT: LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS". ALBERT uses cross-layer parameter sharing to improve model performance. I don't ...
2
votes
1answer
42 views

Machine Translation: with sufficient parallel data, can we improve even further the performance of the system with the use of monolingual data?

I am trying to find scientific literature that studies if, in a situation in which we already have enough parallel data, the addition of monolingual data can further improve performance. I have not ...
0
votes
1answer
43 views

Can a deep learning model learn to understand/interpret the logical sentences?

I don't know what the development in NLP has gone so far. I'm curious about whether my machine can answer me following question? "Today is Sunday. I'll finish my project in two days. What will be day ...
0
votes
1answer
55 views

get predictability of word given sentence in python

I have a paragraph and I want to get the probability (p(word | context) ) of each word, given previous words, for various models (e.g. pre-trained LSTM). Where can pretrained models would allow me to ...
1
vote
1answer
37 views

Do recurrent neural language models greedily model language probability?

Want to check my understanding of recurrent neural language models (in this case I'm working with a decoder in an encoder-decoder RNN but I don't think that matters significantly). I'm trying to ...
4
votes
1answer
578 views

Why can't standard conditional language models be trained left-to-right *and* right-to-left?

From the BERT paper: Unfortunately, standard conditional language models can only be trained left-to-right or right-to-left, since bidirectional conditioning would allow each word to indirectly “...
0
votes
0answers
12 views

How can an untrusted feature value affect the accuracy of a machine learning model?

I'm building a language detector model to classify between 2 languages given text image. The input will be the image, but I have another feature that can tell somehow the language (between 0,1) ...
1
vote
0answers
24 views

Contextual word embeddings to estimate likelihood of word given previous words in sentence?

I'd like to use contextual embeddings to estimate the likelihood of word n given the previous n-1 words in a sentence. Which pretrained models would allow me to do this (could I use something like ...
1
vote
0answers
220 views

Perplexity for short sentences

I have a model that outputs short sentences and want to compare the quality of its outputs for different configurations by computing their perplexities using another model. I tried to use the gpt-2 ...
0
votes
1answer
45 views

Neural language model: Derivation of MLE

Recently, I studied NNLM and I saw the derivation of softmax using MLE: \begin{align} & \frac{\partial\log P(w_t\mid h)}{\partial\theta} \\[8pt] = {} & \frac{\partial \log \exp(s_\theta(w_t,...
3
votes
1answer
204 views

Denoising autoencoders vs masked approaches

I am not an expert in language modelling domain. There are mainly two approaches that are being used nowadays. Denoising autoencoders and ...
0
votes
1answer
72 views

How to train a RNN language model?

I want to train a RNN-based language model from https://arxiv.org/pdf/1409.2329.pdf for next word prediction. How to split the sentences from the dataset into input and ground truth during the ...
0
votes
1answer
49 views

Difference between ELMo and a normal deep network

I was reading up about ELMo and what I could gather was that we essentially combine the weights from different stacked lstm/gru layers for a given token as different layers are suited for different ...
0
votes
1answer
50 views

Likelihood for a test data (sequence of characters) given two unigram models

I would like to find the likelihood of a sequence of characters (the test data), given two unigram models. The sequence (test data) is: A B C B B The models ...
0
votes
1answer
23 views

Modeling words in a language based on their characters

I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
0
votes
1answer
52 views

CMUDict normalized for word frequency

I am trying to train a neural grapheme to phoneme (G2P) model on CMUDict, but I find that pretty soon its loss is barely decreasing. Also, when I train the model on a different similar-sized dataset (...
1
vote
1answer
366 views

How to sample a language model?

I've successfully trained a language model using LSTMs. But I have a confusion about sampling. On sampling, we generate a probability distribution at each time step. It will be of length vocabulary ...
1
vote
1answer
331 views

Where can I find pre-trained language models in English and German?

Where can I find (more) pre-trained language models? I am especially interested in neural network based models for English and German. I am aware only of Language Model on One Billion Word Benchmark ...
0
votes
1answer
308 views

Perplexity of a Non-Statistical Language Model

I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception ...
2
votes
1answer
3k views

How to handle big vocabulary size with keras tokenizer?

I am actually working on a neural language model developed with keras. I have an encoder and a decoder and the output of the decoder is a dense vector on the vocabulary..so quite big depending on the ...
1
vote
1answer
153 views

How does clustering improve a language model?

This article describes a hierarchical clustering algorithm which clusters the words within a vocabulary based on their similarity, in order to improve a language model (in the article, n-grams). How ...
3
votes
0answers
37 views

natural language understanding algorithms [closed]

I am doing some research into how smart personal assistants work like siri, alexa etc. I have found that it is using automatic speech recognition to turn the speech into weighted text form and then ...
0
votes
0answers
92 views

Normalize probability distribution by variance of each class

I have several topics that I will collect language data for. Using Mturk I will ask responders to write sentences for each topic. The sentences will be used to train a language model. Language models ...
2
votes
2answers
71 views

What are commonly used methods to represent a document by a vector?

Methods that I know of Bag of words + weighting: tf-idf, bm25 Topic models: LSA, LDA Word/sentence/document embedding Are there other commonly used methods to represent a document by a vector?
1
vote
1answer
135 views

How to compute context-independent token representations in a biLM?

I've been reading this paper on ELMo word representations. For context, here's my understanding of the standard bi-directional language model (biLM) thus far: Given a sequence of tokens $(t_{1}, ...
1
vote
1answer
28 views

What the 'N' means on the following N-gram approximation

What the 'N'(capital) means in the following N-gram approximation to the conditional probability of the next word in a sequence ?
1
vote
0answers
27 views

Character level RNN for converting word forms

I want to build a char RNN to convert word form from one to another, for example, singular nouns such as lion to lions. However ...
0
votes
1answer
954 views

Perplexity calculation with neural nets

I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is ...
4
votes
1answer
1k views

Skip-gram algorithm confusion

As a newbie to NLP, I am (deeply) confused by the middle step in the following diagram explaining the skip-gram algorithm. The video where this diagram was presented can be found at: https://www....
2
votes
1answer
146 views

How do we pass data to a RNN?

Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we: Scheme 1 Take the first batch of ...