All Questions
Tagged with language-models neural-networks
31 questions
2
votes
1
answer
92
views
What's the loss that is optimized in InstructGPT RL stage?
In the InstructGPT paper they define objective of RL stage as:
They try to maximize this objective using PPO.
I have trouble understanding how they plug this objective into the PPO though. Do they ...
- 143
0
votes
0
answers
248
views
Is it necessary to use attention mask for mean pooling for BERT?
I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
- 138
7
votes
3
answers
2k
views
How do you add knowledge to LLMs?
I recently heard an interesting comment from a gentleman on YouTube and it made sense instantly.
To paraphrase he explained that "fine-tuning" an LM is not necessarily adding knowledge to a ...
- 768
1
vote
1
answer
94
views
Intuitive difference between NN and attention for text prediction
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:
Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
0
votes
1
answer
34
views
Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)
I understand how later neural language models (such as those used in the Word2Vec papers) framed the language modelling problem in a self-supervised way by learning to predict the next word (or any ...
- 1,046
1
vote
1
answer
340
views
Feedforward Neural Net Language Model - computational complexity (word2vec)
While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM):
It consists of input, projection, hidden and output layers. At the ...
- 41
1
vote
1
answer
468
views
How to handle out of the bag token in NLP?
In my current language model my model is unaware of any token that is out-of-bag
for example:- In my summary generating model when we pass some token that is out-of-bag then my model will completely ...
- 13
8
votes
1
answer
573
views
Why are Transformers "suboptimal" for language modeling but not for translation?
Language Models with Transformers states:
Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to ...
- 1,347
2
votes
3
answers
791
views
Confusion about CBOW and Skip-Gram models?
I've read a couple online description of CBOW and Skip-Gram and usually the descriptions starts like this:
We need to train models on words
So we encode words using vectors
One-hot encoding is not ...
- 1,630
1
vote
1
answer
1k
views
Cross layer parameter sharing in ALBERT Model
I am reading the paper "ALBERT: LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS". ALBERT uses cross-layer parameter sharing to improve model performance. I don't ...
- 173
0
votes
1
answer
80
views
Can a deep learning model learn to understand/interpret the logical sentences?
I don't know what the development in NLP has gone so far. I'm curious about whether my machine can answer me following question?
"Today is Sunday. I'll finish my project in two days. What will be day ...
- 155
1
vote
1
answer
86
views
Do recurrent neural language models greedily model language probability?
Want to check my understanding of recurrent neural language models (in this case I'm working with a decoder in an encoder-decoder RNN but I don't think that matters significantly). I'm trying to ...
- 13
0
votes
0
answers
13
views
How can an untrusted feature value affect the accuracy of a machine learning model?
I'm building a language detector model to classify between 2 languages given text image. The input will be the image, but I have another feature that can tell somehow the language (between 0,1) ...
- 101
1
vote
0
answers
31
views
Contextual word embeddings to estimate likelihood of word given previous words in sentence?
I'd like to use contextual embeddings to estimate the likelihood of word n given the previous n-1 words in a sentence. Which pretrained models would allow me to do this (could I use something like ...
- 11
5
votes
1
answer
1k
views
Denoising autoencoders vs masked approaches
I am not an expert in language modelling domain.
There are mainly two approaches that are being used nowadays. Denoising autoencoders and ...
- 378
0
votes
1
answer
62
views
Difference between ELMo and a normal deep network
I was reading up about ELMo and what I could gather was that we essentially combine the weights from different stacked lstm/gru layers for a given token as different layers are suited for different ...
- 101
1
vote
1
answer
384
views
Where can I find pre-trained language models in English and German?
Where can I find (more) pre-trained language models? I am especially interested in neural network based models for English and German.
I am aware only of Language Model on One Billion Word Benchmark ...
3
votes
1
answer
4k
views
How to handle big vocabulary size with keras tokenizer?
I am actually working on a neural language model developed with keras.
I have an encoder and a decoder and the output of the decoder is a dense vector on the vocabulary..so quite big depending on the ...
- 31
2
votes
1
answer
189
views
How to compute context-independent token representations in a biLM?
I've been reading this paper on ELMo word representations.
For context, here's my understanding of the standard bi-directional language model (biLM) thus far:
Given a sequence of tokens $(t_{1}, ...
1
vote
0
answers
35
views
Character level RNN for converting word forms
I want to build a char RNN to convert word form from one to another, for example, singular nouns such as lion to lions. However ...
0
votes
1
answer
1k
views
Perplexity calculation with neural nets
I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is ...
- 101
5
votes
1
answer
1k
views
Skip-gram algorithm confusion
As a newbie to NLP, I am (deeply) confused by the middle step in the following diagram explaining the skip-gram algorithm. The video where this diagram was presented can be found at:
https://www....
- 377
3
votes
1
answer
220
views
How do we pass data to a RNN?
Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we:
Scheme 1
Take the first batch of ...
- 2,693
3
votes
2
answers
453
views
Language Modelling using Neural Networks
I plan to make a Language Model in Python using Neural Networks. I've read that Neural Networks need vectors as input. One common vector representation in NLP is the Bag of Words model. Given a corpus ...
- 53
2
votes
0
answers
451
views
Deciding on the training sequences for RNN/LSTM language model
In a character language model, text is seen as a stream of characters. Say we have a training text as a string s, with length ...
- 692
2
votes
0
answers
2k
views
Computing Training Set Perplexity of a Neural Language Model: Too low values
I am implementing a Language Model based on a Deep Learning architecture (RNN+Softmax).
The cost function I am using is the cross-entropy between the vector of probabilities at the softmax layer and ...
- 71
7
votes
2
answers
3k
views
Calculating test-time perplexity for seq2seq (RNN) language models
To compute the perplexity of a language model (LM) on a test sentence $s=w_1,\dots,w_n$ we need to compute all next-word predictions $P(w_1), P(w_2|w_1),\dots,P(w_n|w_1,\dots,w_{n-1})$.
My question ...
- 96
5
votes
1
answer
8k
views
Advantage of character based language models over word based
Is there an intuition why character based models language bases models are preferred over word based. For example Karpathy builds his language model by predicting the next character in Karpathy Blog.
...
- 211
1
vote
1
answer
810
views
Does the skipgram language model try to predict all context words at the same time?
In the skipgram language model (Mikolov et al., 2013), a neural network with one hidden layer tries to predict surrounding words from current words of the corpus. After training, the hidden activation ...
- 990
2
votes
1
answer
937
views
Why two layers of matrix multiplication is needed in CBOW and Skip-gram model?
I found a nice tutorial here regarding CBOW and Skip-gram models for Word2Vec. I got the following doubt:
Why two layers of matrix multiplication is need for CBOW and SKIP-GRAM ? As multiplying a ...
- 153
11
votes
2
answers
3k
views
Question about Continuous Bag of Words
I'm having trouble understanding this sentence:
The first proposed architecture is similar to the feedforward NNLM,
where the non-linear hidden layer is removed and the projection
layer is ...
- 323