Questions tagged [natural-language]

Natural Language Processing is a set of techniques from linguistics, artificial intelligence, machine learning and statistics that aim at processing and understanding human languages.

Filter by
Sorted by
Tagged with
3 votes
2 answers
209 views

What exactly is meant by isotropic and anisotropic with word vectors

From this paper https://aclanthology.org/D19-1006.pdf "How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings" When they say word ...
  • 112
0 votes
0 answers
5 views

Does the concept of "confidence" apply to text generation tasks?

I'm currently using the T5 model to solve a text classification task as text generation. Specifically, I'm using the T5ForConditionalGeneration model from the ...
  • 2,649
0 votes
0 answers
8 views

How do you measure a "confidence score" for text generation?

For typical classification tasks my understanding is that you simply take the output logits of the model, convert them to a probability score via softmax or something, and then use the score of the ...
  • 2,649
1 vote
1 answer
21 views

How to avoid underflow of the probability of sentence in calculating the perplexity of corpus

I am looking at this post How to find the perplexity of a corpus. I understand the whole post, but the probability of a sentence appear in a corpus, in a unigram model, is given by p(s)=∏ni=1p(wi), ...
  • 13
0 votes
0 answers
5 views

Text classification with sequences containing two languages?

I'm currently trying to make a text classification model which receives a text sequence containing more than one (two in my case) language. I've tried using multilingual language models (e.g., ...
  • 2,649
1 vote
0 answers
29 views

Skip gram model and negative log loss likelihood

I recently just started learning about NLP and word representation. I have been trying to implement the negative log loss likelihood function but am having some trouble with it and would like to ask ...
1 vote
0 answers
9 views

How to interpret the relationship between batch size and bootstrap count in a specific paper?

In the paper "Active Learning for Natural Language Parsing and Information Extraction", the author mentioned: In tests on this data, test examples were chosen independently for 10 trials ...
  • 79
1 vote
0 answers
14 views

processing natural language that descripe time frequency with R

I'm dealing with data that descripe onset frequency of a symptom. The text in each cell was not in the same format. For example: ...
0 votes
0 answers
12 views

How do I interpret the results of laplace smoothed bigram?

I've applied laplace smoothing on creating bigram model of a piece of text in Python, but I'm unsure as to how to interpret the results. From what I understand, laplace smoothing is add-1 smoothing, ...
  • 1
0 votes
0 answers
14 views

Why does the masked language modelling (MLM) task produce useful embeddings?

Masked language modelling is the standard way of training a language model such as a transformer. Each input token has some probability (e.g. 15%) of being replaced with a <MASK> token. The ...
  • 721
0 votes
0 answers
18 views

Transition-Based Chunking for NER Task

I am trying to understand how the spacy.TransitionBasedParser.v2 architecture works in SpaCy when running custom training to run a NER Task. After doing a lot of research on the web, I found that this ...
2 votes
2 answers
61 views

Detect quoted text in emails

I have small dataset (<10k) of emails (plaintext) that need to be classified. Currently I'm doing research on topic of email preprocessing and I can't find any suitable solution for quoted text ...
  • 23
1 vote
0 answers
16 views

Intuitive explanation for summing the embedding and positional encoding in the Transformer's embedding

In the Transformer model, the embedding and positional encoding are summed together to represent a word in each location ('positional embedding' from now on). This way, each cell contains semantic and ...
  • 301
0 votes
0 answers
12 views

How to train a language model (like BERT, or any other) to not classify a sentence in a particular class?

I have limited data, around 50 sentences per class and around 70 classes. Due to this reason of having less data, sometimes some sentences get classified in a class, which are similar but not ...
0 votes
0 answers
20 views

Postprocessing word vectors before visualization [duplicate]

I'm studying the work2vec code from stanfords's NLP course CS224n. After training the word vectors, they perform the following operations ...
0 votes
0 answers
25 views

Doesn't positional encoding create noise in embedding(features) of word?

The purpose of embedding is to convert words into numbers. The key to solving any NLP problem requires precise embedding, but during Transformer I got familiar with concept positional encoding. And ...
0 votes
0 answers
8 views

Neural Network Feeding - Custom Information Extraction

I am trying to train a neural net. to extract specific information from text. I need to find entity information like attribute, dependency, etc. The text input will be like this: ...
4 votes
1 answer
51 views

Why do large LMs use the transpose of the word embeddings matrix in the classification head?

All literature, guides and tutorials describing the construction of language models have used two separate matrices for the input and output projections: To project one-hot token IDs into hidden ...
  • 1,076
1 vote
1 answer
27 views

Do I need training data in multiple languages for a multilingual transformer?

I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case. ...
  • 333
0 votes
0 answers
12 views

Comparing two LDA models from similar corpuses on political social media posts

I am working with political data from the 2020 elections. My first LDA topic model is about 17500 docs and focused on the overall political data. The second LDA model I am going to run will be using ...
0 votes
0 answers
28 views

Transformers for correcting single word misspellings

I'm asking for your kind help to know if there is some known strategy/reference to use a transformer-like model to solve the following problem: The input is a single misspelled word, such as: $$dta$$ ...
0 votes
0 answers
7 views

Ways to detect noise in multi-class classification training data using text embeddings (BERT)

So I have a dataset with a column of text and and labels (5 different labels) associated with it. The labels describe the potential answer to the type of question being asked in the text column. For ...
2 votes
1 answer
23 views

How to separate text into two different parts (positive/negative) without hardcoded rules?

The task is simply explainable but I imagine very hard. I have a large (1000+) recordings from a questionnaire in which people are asked to describe some of their good and bad personality traits and ...
1 vote
0 answers
106 views

How can I use Word Embeddings with Naive Bayes to get most important feature in text classification?

Let's say I have a corpus of news articles, for which I want to classify into various topics (e.g. "Entertainment", "Tech", "Science") and obtain the most predictive ...
2 votes
1 answer
16 views

"General" domain hyperparameter tuning across classes

I am conducting experiments which requires me to create a somewhat well performing model while avoiding (as much as possible) the computational expensive search over sets of hyperparameters. My ...
0 votes
0 answers
10 views

Why does the formula of IDF in TD-IDF technique use log? [duplicate]

I understand that the idea of IDF is to measure the importance of a term across a corpus by weighing down the terms that are very common across a corpus and weighing up the rare ones. Formula: $$IDF(...
  • 111
1 vote
0 answers
24 views

NLP - How to deal with a dataset where some spaces between words are missing

I've been normalizing a dataset and after tokenizing my words I've noticed that some records contain combinations of words where the spaces between them are missing. ...
  • 111
0 votes
0 answers
17 views

A special case for text matching which answers whether a sentence is subsumed by another

Semantic matching for sentence pairs has been well studied and applied in many NLP applications such as natural language inference (NLI) and question answering (QA). In different applications, ...
  • 153
0 votes
0 answers
17 views

What is masking doing to the padding in the encoder stage of transformer? [duplicate]

I'm referring to this post What I don't understand is, if the result of padding is simply adding zeroes to ensure all input sequences are of the same length what is the masking stage doing exactly ? ...
1 vote
1 answer
302 views

How to use cosine similarity within triplet loss

The triplet loss is defined as follows: $$ L(A, P, N) = max(‖f(A) - f(P)‖² - ‖f(A) - f(N)‖² + margin, 0) $$ where $A$=anchor, $P$=positive, and $N$=negative are the data samples in the loss, and $...
  • 541
0 votes
0 answers
8 views

How can I evaluate my work if there is no benchmark study for my targeted domain?

I have some questions if possible... My project is to create a model to detect fake news in a specific domain, which has not been investigated in this specific domain by previous studies. Data on this ...
1 vote
1 answer
26 views

Is there a rule of thumb (or actual numbers) for the minimal number of words in a document for TFiDF to work?

I'm not a data scientist, and googling for it lead me to thinking I'm not asking the question correctly. With a high risk of this being closed as duplicate (which will be great, as it will help me ...
0 votes
0 answers
13 views

Is GloVe really learn co-occurrence probabilities ratio (Pik/Pjk in paper) rather than the probabilities themselves?

In the cost function, there is only the co-occurrence, not even co-occurrence probabilities.
  • 1
1 vote
0 answers
15 views

Content Based Recommendation System

how will a content-based recommendation system recommend after a sudden change in taste of user?? what will the system recommend to new users without any data about them available in content-based ...
0 votes
0 answers
9 views

Am I understanding training a model to use in a similar task (both translations but from different language pairs) later, wrong?

I am currently training an mt5 in Spanish to English (and vice versa) translation. That works with a bleu of 40ish. (both ways). I then want to use that same model I trained to improve the translation ...
0 votes
0 answers
10 views

Why does latent dirichlet allocation (LDA) fail when dealing with large and heavy-tailed vocabularies?

I'm reading the 2019 paper Topic Modeling in Embedding Spaces which claims that the embedded topic model improves on these limitations of LDA. But why does LDA have these limitations—why does it fail ...
2 votes
0 answers
33 views

Do Transformers pad input sentences?

Assume we have a Transformer (Attention is all you need paper) and we give to it an input sequence S of length $n_{words}$. If no padding is applied, the output of the encoder model would be a matrix ...
  • 31
1 vote
0 answers
23 views

Does average or max pooling actually summarise the sentence?

I am working on an multi-label text classification problem at work and adapted model architecture from this notebook of Toxic Comment Classification challenge on Kaggle. I have trained the model, a ...
0 votes
0 answers
82 views

How to calculate euclidian distance from similarity matrix

I have a similarity matrix but can not use it in the k-mean as input. So is there any way to convert the similarity matrix to the euclidean distance matrix? Edit: I have Jaccard, overlap, cosine,(...
0 votes
0 answers
14 views

Training loss first fall then rise when training a bidirectional LSTM model for a NER task

I'm training a bidirectional-LSTM (Bi-LSTM) + conditional random field (CRF) model for a named entity recognition task. The training set contains 7033 labeled sentences, and the validation set for ...
0 votes
0 answers
17 views

Text Classification of documents with high variance in lengths

I want to categorize text documents, which have a length between 25 and 3000 words. Language models like BERT only support 512 tokens. It looks like sometimes all 3000 words are needed semantically. I ...
  • 1,149
0 votes
0 answers
11 views

cross-domain analysis fake news detection

I am new to machine learning and I am doing a project for my dissertation on fake news detection. I am doing a cross-domain analysis. Training classifier on one dataset and testing it on another ...
1 vote
0 answers
10 views

Using difference of sentences as concatenation for sentence similarity task

MT-DNN and SentenceBERT concatenate sentences with their difference for sentence similarity task (i.e. $[u; v; u-v]$). However, I can not find any sources, backgrounds or experiment results for ...
2 votes
0 answers
242 views

How long to wait for Bert accuracy to improve during training

I have a very difficult binary text classification problem for which I am trying to fine-tune a bert model (bert_en_uncased_L-12). By "difficult" I mean, the text data contains a large ...
  • 21
2 votes
0 answers
35 views

How to tell model (Multiclass Classification using Logistic Regression) is overfitting?

I'm training a logistic classifier to classify 5 classes using scikit-learn. The data isn't extremely imbalanced (class 1: 27.7%, class 2: 19.4%, class 3: 17%, class 4: 19.6%, class 5: 16.2%). I'm ...
  • 21
0 votes
0 answers
21 views

Why does PCA on high-dimensional text data improve the performance of random forest but not a decision tree classifier?

Why does PCA on high-dimensional text data improve the performance of random forest but not a decision tree classifier? I am dealing with high dimensional text data. I found that performing PCA to an ...
  • 1
1 vote
0 answers
24 views

References for training BERT-like models from scratch

As the title of the question suggests, I'm interested in training a BERT-like model (and then use it to make some experiments on text-similarity). Question: Could you share some references on the ...
1 vote
1 answer
29 views

Reliable methods to validate clustering of text phrases?

Question is in the title. I have clustered the word embeddings of text phrases, and now want to try and check whether the resulting clusters are coherent enough. I have tried methods that are ...
0 votes
0 answers
16 views

Classifying words/phrases that say the same thing in slightly different ways

I have survey data in which participants can manually write one or more skills that they possess. Many responses are similar enough to be classified as the same but are either not identical (e.g., &...
1 vote
2 answers
393 views

Why not perform weight decay on layernorm/embedding?

I am learning the code of minGPT. In the function, the author excluded layernorm and embedding layer from experiencing weight decay and I want to know the reasons. Besides, what about batchnorm?

1
2 3 4 5
23