All Questions
Tagged with language-models machine-learning
29 questions
1
vote
1
answer
91
views
How can we implement pipelines using LLMs that can consider information from the internet in their responses?
Let us suppose that we are implementing a data processing pipeline based on LLMs and, in some part of the process, we need to search the internet for finding relevant information regarding some topic, ...
- 1,008
0
votes
0
answers
296
views
Bert Used for generative AI
I have a doubt regarding using "Bert" as a generative model. I know Bert can be used for classification or fine-tuning the question-answering. However, is it possible to use Bert to generate ...
- 175
5
votes
1
answer
1k
views
Should a language model like GPT-3 be directly used to perform classification?
The OpenAI API enables classification by sampling from GPT-3 given a prompt. Is estimating posterior probabilities a more statistically sound approach? Below is a specification of what "...
- 1,625
2
votes
1
answer
403
views
Do I need training data in multiple languages for a multilingual transformer?
I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.
...
- 465
1
vote
1
answer
468
views
How to handle out of the bag token in NLP?
In my current language model my model is unaware of any token that is out-of-bag
for example:- In my summary generating model when we pass some token that is out-of-bag then my model will completely ...
- 13
2
votes
0
answers
622
views
Variable batch size for inputs of different length
We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
2
votes
1
answer
349
views
Does the transformers model (in “Attention is All You Need”) exclude the encoder in language modelling tasks?
The language model I am referring to is the one outlined in "Attention is All You Need":
My understanding is (please correct me if I am wrong) that when the task is translation, the encoder'...
- 23
3
votes
1
answer
187
views
Breaking substitution cipher with language model
Frequency analysis is a common tool used to break substitution ciphers, but often relies on intuition and guesswork of a human. Since language models can objectively calculate perplexity (how ...
3
votes
1
answer
4k
views
Use of ignore_index on CrossEntropyLoss() for text models
I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this.
From what I understand ...
- 33
1
vote
0
answers
38
views
Language Model dealing with Book Dialogue
My goal is to generate text based on a specific book using an LSTM language model. One problem with my generator is that it seems that the book's dialogue is somewhat messing up my generator. I had ...
5
votes
2
answers
1k
views
Generating text from language model
I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be:
Apply softmax function
Take a weighted random choice to determine next word
This ...
0
votes
0
answers
13
views
How can an untrusted feature value affect the accuracy of a machine learning model?
I'm building a language detector model to classify between 2 languages given text image. The input will be the image, but I have another feature that can tell somehow the language (between 0,1) ...
- 101
5
votes
1
answer
1k
views
Denoising autoencoders vs masked approaches
I am not an expert in language modelling domain.
There are mainly two approaches that are being used nowadays. Denoising autoencoders and ...
- 378
0
votes
1
answer
62
views
Difference between ELMo and a normal deep network
I was reading up about ELMo and what I could gather was that we essentially combine the weights from different stacked lstm/gru layers for a given token as different layers are suited for different ...
- 101
0
votes
1
answer
27
views
Modeling words in a language based on their characters
I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
- 13
0
votes
1
answer
519
views
Perplexity of a Non-Statistical Language Model
I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception ...
- 101
3
votes
0
answers
39
views
natural language understanding algorithms [closed]
I am doing some research into how smart personal assistants work like siri, alexa etc. I have found that it is using automatic speech recognition to turn the speech into weighted text form and then ...
- 31
2
votes
2
answers
120
views
What are commonly used methods to represent a document by a vector?
Methods that I know of
Bag of words + weighting: tf-idf, bm25
Topic models: LSA, LDA
Word/sentence/document embedding
Are there other commonly used methods to represent a document by a vector?
- 16.8k
5
votes
1
answer
1k
views
Skip-gram algorithm confusion
As a newbie to NLP, I am (deeply) confused by the middle step in the following diagram explaining the skip-gram algorithm. The video where this diagram was presented can be found at:
https://www....
- 377
0
votes
0
answers
75
views
Decide threshold for LM to filter out ill-formed sentence
I have a corpus of sentences. This corpus contains well-formed as well as ill-formed sentences. I want to filter out ill-formed sentences from the corpus.
Ill-formed sentences are of two types :
...
3
votes
2
answers
453
views
Language Modelling using Neural Networks
I plan to make a Language Model in Python using Neural Networks. I've read that Neural Networks need vectors as input. One common vector representation in NLP is the Bag of Words model. Given a corpus ...
- 53
3
votes
2
answers
2k
views
Language Detection with CLD2 with Mixed Inputs in long documents
Internals Recap. CLD2 is a Naïve Bayesian classifier, trained on documents of mean size of 200 characters, trained on a corpus of 100M scraped and human expert selected web pages.
When working on ...
- 153
2
votes
1
answer
6k
views
How to calculate the perplexity of test data versus language models
I have been working on an assignment where I train upon 3 corpora in 3 separate languages, and then I read in a set of sentences and use a number of models to determine the most likely language for ...
- 173
5
votes
2
answers
12k
views
Understanding Add-1/Laplace smoothing with bigrams
I am working through an example of Add-1 smoothing in the context of NLP
Say that there is the following corpus (start and end tokens included)
...
- 173
5
votes
1
answer
8k
views
Advantage of character based language models over word based
Is there an intuition why character based models language bases models are preferred over word based. For example Karpathy builds his language model by predicting the next character in Karpathy Blog.
...
- 211
2
votes
1
answer
145
views
How many words does the algorithm search through in Google Ngram? [closed]
When I run a query for "hers" in Google Ngram Viewer, I get back the word's frequency of occurrence as a percentage. We know the outcome percentage; what's the denominator on the other size? Is it 100 ...
- 646
11
votes
2
answers
3k
views
Question about Continuous Bag of Words
I'm having trouble understanding this sentence:
The first proposed architecture is similar to the feedforward NNLM,
where the non-linear hidden layer is removed and the projection
layer is ...
- 323
16
votes
3
answers
5k
views
In Kneser-Ney smoothing, how are unseen words handled?
From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as
$
\begin{align}
P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
- 311
11
votes
3
answers
16k
views
Regarding using bigram (N-gram) model to build feature vector for text document
A traditional approach of feature construction for text mining is bag-of-words approach, and can be enhanced using tf-idf for setting up the feature vector characterizing a given text document. At ...
- 3,089