Skip to main content

All Questions

Filter by
Sorted by
Tagged with
2 votes
1 answer
91 views

What's the loss that is optimized in InstructGPT RL stage?

In the InstructGPT paper they define objective of RL stage as: They try to maximize this objective using PPO. I have trouble understanding how they plug this objective into the PPO though. Do they ...
Druudik's user avatar
  • 143
0 votes
0 answers
248 views

Is it necessary to use attention mask for mean pooling for BERT?

I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
mutli-arm-bandit's user avatar
7 votes
3 answers
2k views

How do you add knowledge to LLMs?

I recently heard an interesting comment from a gentleman on YouTube and it made sense instantly. To paraphrase he explained that "fine-tuning" an LM is not necessarily adding knowledge to a ...
Edv Beq's user avatar
  • 768
1 vote
1 answer
94 views

Intuitive difference between NN and attention for text prediction

If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things: Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
tunafriedrice's user avatar
0 votes
1 answer
34 views

Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)

I understand how later neural language models (such as those used in the Word2Vec papers) framed the language modelling problem in a self-supervised way by learning to predict the next word (or any ...
Felipe's user avatar
  • 1,046
1 vote
1 answer
340 views

Feedforward Neural Net Language Model - computational complexity (word2vec)

While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM): It consists of input, projection, hidden and output layers. At the ...
MDescamps's user avatar
1 vote
1 answer
468 views

How to handle out of the bag token in NLP?

In my current language model my model is unaware of any token that is out-of-bag for example:- In my summary generating model when we pass some token that is out-of-bag then my model will completely ...
Nervous Hero's user avatar
8 votes
1 answer
573 views

Why are Transformers "suboptimal" for language modeling but not for translation?

Language Models with Transformers states: Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to ...
MWB's user avatar
  • 1,347
2 votes
3 answers
790 views

Confusion about CBOW and Skip-Gram models?

I've read a couple online description of CBOW and Skip-Gram and usually the descriptions starts like this: We need to train models on words So we encode words using vectors One-hot encoding is not ...
Fraïssé's user avatar
  • 1,630
1 vote
1 answer
1k views

Cross layer parameter sharing in ALBERT Model

I am reading the paper "ALBERT: LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS". ALBERT uses cross-layer parameter sharing to improve model performance. I don't ...
Zephyr's user avatar
  • 173
0 votes
1 answer
80 views

Can a deep learning model learn to understand/interpret the logical sentences?

I don't know what the development in NLP has gone so far. I'm curious about whether my machine can answer me following question? "Today is Sunday. I'll finish my project in two days. What will be day ...
Ruchit Patel's user avatar
1 vote
1 answer
86 views

Do recurrent neural language models greedily model language probability?

Want to check my understanding of recurrent neural language models (in this case I'm working with a decoder in an encoder-decoder RNN but I don't think that matters significantly). I'm trying to ...
Brendan King's user avatar
0 votes
0 answers
13 views

How can an untrusted feature value affect the accuracy of a machine learning model?

I'm building a language detector model to classify between 2 languages given text image. The input will be the image, but I have another feature that can tell somehow the language (between 0,1) ...
shrouk mansour's user avatar
1 vote
0 answers
31 views

Contextual word embeddings to estimate likelihood of word given previous words in sentence?

I'd like to use contextual embeddings to estimate the likelihood of word n given the previous n-1 words in a sentence. Which pretrained models would allow me to do this (could I use something like ...
mchlmchl's user avatar
5 votes
1 answer
1k views

Denoising autoencoders vs masked approaches

I am not an expert in language modelling domain. There are mainly two approaches that are being used nowadays. Denoising autoencoders and ...
enterML's user avatar
  • 378
0 votes
1 answer
62 views

Difference between ELMo and a normal deep network

I was reading up about ELMo and what I could gather was that we essentially combine the weights from different stacked lstm/gru layers for a given token as different layers are suited for different ...
shiredude95's user avatar
1 vote
1 answer
384 views

Where can I find pre-trained language models in English and German?

Where can I find (more) pre-trained language models? I am especially interested in neural network based models for English and German. I am aware only of Language Model on One Billion Word Benchmark ...
3 votes
1 answer
4k views

How to handle big vocabulary size with keras tokenizer?

I am actually working on a neural language model developed with keras. I have an encoder and a decoder and the output of the decoder is a dense vector on the vocabulary..so quite big depending on the ...
pokatore's user avatar
2 votes
1 answer
189 views

How to compute context-independent token representations in a biLM?

I've been reading this paper on ELMo word representations. For context, here's my understanding of the standard bi-directional language model (biLM) thus far: Given a sequence of tokens $(t_{1}, ...
infinitely_improbable's user avatar
1 vote
0 answers
35 views

Character level RNN for converting word forms

I want to build a char RNN to convert word form from one to another, for example, singular nouns such as lion to lions. However ...
Abhishek Malpani's user avatar
0 votes
1 answer
1k views

Perplexity calculation with neural nets

I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is ...
Boris Mocialov's user avatar
5 votes
1 answer
1k views

Skip-gram algorithm confusion

As a newbie to NLP, I am (deeply) confused by the middle step in the following diagram explaining the skip-gram algorithm. The video where this diagram was presented can be found at: https://www....
MeiNan Zhu's user avatar
3 votes
1 answer
220 views

How do we pass data to a RNN?

Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we: Scheme 1 Take the first batch of ...
figs_and_nuts's user avatar
3 votes
2 answers
453 views

Language Modelling using Neural Networks

I plan to make a Language Model in Python using Neural Networks. I've read that Neural Networks need vectors as input. One common vector representation in NLP is the Bag of Words model. Given a corpus ...
Jessie's user avatar
  • 53
2 votes
0 answers
451 views

Deciding on the training sequences for RNN/LSTM language model

In a character language model, text is seen as a stream of characters. Say we have a training text as a string s, with length ...
Yibo Yang's user avatar
  • 692
2 votes
0 answers
2k views

Computing Training Set Perplexity of a Neural Language Model: Too low values

I am implementing a Language Model based on a Deep Learning architecture (RNN+Softmax). The cost function I am using is the cross-entropy between the vector of probabilities at the softmax layer and ...
ML_TN's user avatar
  • 71
7 votes
2 answers
3k views

Calculating test-time perplexity for seq2seq (RNN) language models

To compute the perplexity of a language model (LM) on a test sentence $s=w_1,\dots,w_n$ we need to compute all next-word predictions $P(w_1), P(w_2|w_1),\dots,P(w_n|w_1,\dots,w_{n-1})$. My question ...
xhi's user avatar
  • 96
5 votes
1 answer
8k views

Advantage of character based language models over word based

Is there an intuition why character based models language bases models are preferred over word based. For example Karpathy builds his language model by predicting the next character in Karpathy Blog. ...
PKuhn's user avatar
  • 211
1 vote
1 answer
810 views

Does the skipgram language model try to predict all context words at the same time?

In the skipgram language model (Mikolov et al., 2013), a neural network with one hidden layer tries to predict surrounding words from current words of the corpus. After training, the hidden activation ...
danijar's user avatar
  • 990
2 votes
1 answer
937 views

Why two layers of matrix multiplication is needed in CBOW and Skip-gram model?

I found a nice tutorial here regarding CBOW and Skip-gram models for Word2Vec. I got the following doubt: Why two layers of matrix multiplication is need for CBOW and SKIP-GRAM ? As multiplying a ...
aroyc's user avatar
  • 153
11 votes
2 answers
3k views

Question about Continuous Bag of Words

I'm having trouble understanding this sentence: The first proposed architecture is similar to the feedforward NNLM, where the non-linear hidden layer is removed and the projection layer is ...
user70394's user avatar
  • 323