Questions tagged [natural-language]

Natural Language Processing is a set of techniques from linguistics, artificial intelligence, machine learning and statistics that aim at processing and understanding human languages.

Filter by
Sorted by
Tagged with
0
votes
1answer
12 views

Concept drift in text data

Can we detect concept drift in text data?. I am dealing with text classification problem. If we can, how can we detect concept drift in text data?
0
votes
0answers
15 views

NLP: how to quantify information-richeness of short text (i.e., tweet)

I'm not very familliar with NLP or text mining, so forgive me if this is naive. Background I'm working on a personal project, where I fetch tweets from many people and then I try to do some filtering ...
0
votes
0answers
17 views

Pre-requisites for GPT-3

What papers should I read before attempting to understand the GPT-3 paper (apart from the GPT-1 and GPT-2 papers)? Given that GPT-3 is a language model, I'm looking at NLP-related papers specifically. ...
0
votes
0answers
6 views

Transforming topics into text data

I was reading some articles on topic classification, in which some algorithm uses snippets of text as input and tries to classify them in topics, and I thought of implementing this technique in my ...
0
votes
0answers
15 views

Pretrained models for sexual harassment detection

I'm looking for pretrained ready-to-use models for sexual harassment detection in chat messages.. Preferably (but not necessarily) free of charge, open source, and allowing for domain specific fine-...
0
votes
0answers
16 views

product categorization by short strings and one or more powerful factors - a climbers problem

I am using webscraping data to classify products related to the sport of climbing. For all shops i get a product name (which is kind of a short description) plus a category-string as how the product ...
1
vote
0answers
8 views

Why does BERT has a limitation of only allowing the maximum length of the input tokens as 512?

I have seen BERT was one of the state-of-the-arts word embedding method in 2018 and then XLNet is proposed in 2019 to take care of the limitations of BERT. I have seen one limitation of BERT is the ...
0
votes
1answer
13 views

What is the difference between “greedy selection” and “sampling according to a distribution?”

I'm currently studying language generation and had a question regarding some concepts. The paper I'm reading states that they formulate the task of next-token generation as conditionally generating ...
0
votes
0answers
5 views

Training and testing transformer model from scratch

As you know, transformers are one of the strongest model in the field of NLP and machine translation. I know there are many resources, but I still could not find a good tutorial teaching how to use ...
0
votes
1answer
15 views

What classifier could predict spam/ham labels for SMS messages better than Naive Bayes?

I have 7000 SMS messages, 6000 ham, 1000 spam. Typical messages are: ...
0
votes
0answers
26 views

Trying to increase the F1 score for an NLP data extraction problem

We have a problem reaching a decent F1 score when tackling this NLP data extraction problem. When given a group (i.e. Female dogs, male dogs) we want to extract relevant numerical data from a ...
1
vote
0answers
14 views

Neural network architecture that takes in a matrix as input and outputs a vector?

I'm trying to rank words in a sentence by "importance". For example "this is a dog" can have an output like "0.1, 0.2, 0.01, 0.7". The precise numbers don't matter, just ...
1
vote
1answer
15 views

How are the matrices Wq, Wk and Wv calculated in the Transformer?

I am currently reading the "Attention is all you need" paper by Vaswani et.al. The paper says that for each word we generate a query vector, key vector and value vector by multiplying by ...
0
votes
0answers
6 views

When training embeddings should negative samples be distinct from the context?

Suppose I am training word2vec embeddings with skipgrams. I have defined my context and my target word, and now I am looking for negative samples. It just so happens that I randomly sample a word that ...
0
votes
0answers
7 views

Creating a text classification framework based on whether the text agrees with a story or refutes it

I am looking to use nlp to determine whether individual tweets agree with or refute a story based around fake news (i.e. I have a dataset that mentions 5g and coronavirus and I want to know which ...
2
votes
1answer
84 views

In training, I first have a solid drop in loss, but eventually the loss slowly but consistently increases. What could cause this?

I haven't even finished 1 epoch, so I don't think it could any sort of overfitting. I am training on a very large amount of data (27 gb of text) so it'll still be a while before I even reach one epoch....
0
votes
0answers
11 views

Good starting point for NLP sentence classification on limited computational resources?

I am designing a project where I plan to train a supervised ML model to classify English sentences of 10-30 words long into one of 8 or 9 categories by what kind of meaning it conveys. I have a good ...
1
vote
0answers
48 views

GPU vs TPU for convolutional neural networks (NLP) [migrated]

I am testing ideas on IMDB sentiment analysis task by using word embeddings + CNN approach. What could explain a significant difference in computation time in favor of GPU (~9 seconds per epoch) ...
-1
votes
0answers
39 views

A little help on text classification

Right now, I am working on building a efficient classification for my company. We work as a social monitoring company, basically we collect data from social media sites to see the engagement, comment, ...
0
votes
0answers
4 views

Extrinsic Evalutation Peformance of Google Autofill

Does anyone know approximately how successful Google's autofill algorithm is for search completion? Does is successfully return the next word the user was searching for in it's top 5 suggested results ...
0
votes
0answers
13 views

BERT masking - why does it require sampling, and how does it mitigate the mismatch of the [MASK] token when fine-tuning?

I'm reading the BERT paper and jalammar's illustrative guide for BERT. I don't understand 2 things about the method's crux - the masked language model: why does masking requires us to sample (take ...
0
votes
0answers
12 views

fasttext vectors - are they polar or cartesian?

Can anyone explain the meaning of word vectors in fastText? If I run a model with only 3 vectors and then plot them (by treating the 3 vectors as x,y,z Cartesian coordinates) the words seem randomly ...
0
votes
1answer
24 views

Intuition behind the use of multiple attention heads

Consider this introduction to attention layers with the main description below. I understand attention layers as learnable soft query retrieval operators that act on a "K-V store" of vectors....
0
votes
0answers
15 views

Arbitrary threshold for sigmoid activation function for CNN binary classification?

I am classifying sentiment of reviews - 0 or 1 - using gensim Doc2Vec and CNN in ...
0
votes
0answers
13 views

NLP multiclass classification with many sparse classes

I am attempting to use natural language processing to geocode "addresses". The address is the result of a write-in of a survey where the respondent is instructed to give their city, state, ...
0
votes
0answers
7 views

Gensim LDA Topic-Term matrix all Zero

I meet a confusing problem when using gensim.models.ldamodel for topic modeling. I have cleaned my documents set and extract the dictionary as suggested in LDA ...
0
votes
0answers
13 views

Unit of analysis for sentiment analysis

When running sentiment analysis or emotion analysis on social media conversations, is it more correct to analyze sentiment by total word mentions in the corpus or sentiment polarity of individual ...
1
vote
0answers
17 views

Why do transformers use layer norm instead of batch norm?

Both batch norm and layer norm are common normalization techniques for neural network training. I am wondering why transformers primarily use layer norm.
0
votes
1answer
20 views

Attention Mechanisms and Alignment Models in Machine Translation

From the paper that introduced attention mechanisms (Bahdanau et al 2014: Neural Machine Translation by Jointly Learning to Align and Translate), it seems that the translating part is the regular RNN/...
0
votes
0answers
11 views

Performance of model decreases when calculating Euclidean distance between vectors with TF.IDF weigths compared to TF weights

My problem in short: I use Jaccard similarity, cosine similarity and euclidean distance to compute a similarity between documents. The documents consists of either the words,hashtags or combination of ...
0
votes
0answers
15 views

What is GloVe model's loss function actually minimizing?

I am struggling with exactly what is being minimized in the GloVe model. I've read every single blog post, watched every single Youtube video, and some papers that cited GloVe (and of course, read ...
0
votes
2answers
54 views

What is NLP? How it is related to Machine Learning? [closed]

Is NLP a subtopic within machine learning or it is related to Machine Learning? If I want to get started at this, how would I start?
0
votes
0answers
11 views

Named entity recognition class imbalance

I am asking how to treat class imbalance named entity recognition? In fact,the class Other is the dominant class taking into account the fact that we are manipulating sequences (different sentences ...
0
votes
0answers
11 views

Sentiment about topics (defined by LDA)

For a project I do a research about topics in online reviews, and the effect of these topics on the rating (by doing a regression). Furthermore and the reason I would like to have your help: I would ...
8
votes
2answers
341 views

Statistical uncertainties in Deep Learning

Recently I got very interested in NLP applications of deep learning. Diving into literature (on arXiv for instance) I noticed that is very unpopular to quote and estimate uncertainties on scores of ML ...
4
votes
1answer
151 views

No difference in the distribution of data across classes

I'm having a problem in my Deep Learning model where I encounter a class imbalance and there's no virtual difference between the data for the two classes or they have an identical if not similar ...
0
votes
3answers
27 views

Confusion about CBOW and Skip-Gram models?

I've read a couple online description of CBOW and Skip-Gram and usually the descriptions starts like this: We need to train models on words So we encode words using vectors One-hot encoding is not ...
0
votes
0answers
14 views

Specify some words are not important and some words are extra important in topic modelling

I am using Latent Dirichlet Allocation in sklearn to model topics. I have a list of words that are likely to inform topics and a list of words that are not important (like stop words). Is there a way ...
35
votes
4answers
5k views

Is LSTM (Long Short-Term Memory) dead?

From my own experience, LSTM has a long training time, and does not improve performance significantly in many real world tasks. To make the question more specific, I want to ask when LSTM will work ...
2
votes
2answers
44 views

In NLP, what is the difference between corpus and vocabulary? Or are they the same thing?

In NLP, what is the difference between corpus and vocabulary? I see these words often referred to and I feel like they are referring to the same thing. Is there a difference between them or are they ...
0
votes
0answers
13 views

How does structural topic modeling's estimateEffect works?

I've been searching this for a while but haven't been able to find any information regarding how the STM's estimateEffect works. I'm reading the stm package ...
0
votes
0answers
12 views
0
votes
1answer
19 views

Masking BERT subword tokens for NER?

I have the following named-entity recognition (NER) model: (BERT > LSTM > CRF) Extracting BERT vectors needs that we tokenize the input text using BERT tokenizer which tokenizes the text into ...
1
vote
0answers
30 views

Inference time for text genration using fine-tuned gpt2

I have re-trained GPT2 model using over 10 million sentences for QA. And while testing also I am getting very good results. The only problem now I am facing is that I have millions of test data that I ...
0
votes
0answers
7 views

Keras LSTM POS tagger w/ transfer learning (GloVe) — failing to learn?

I've been trying to research how to use Keras to train a POS tagger; specifically I want it to use an LSTM architecture and to use word embeddings, namely, GloVe. I've taken inspiration from two blogs....
0
votes
1answer
14 views

Why do Dense layers perform better than a mix of Conv Layers, Recurrent Layers on Sentiment Analysis with BERT emebddings?

I have used BERT to make embeddings out of the imdb review dataset and I am trying out some models to check their perfomance on sentiment analysis (0 for the bad reviews and 1 for the good ones). I ...
0
votes
0answers
40 views

Normalization of corpus to find perplexity

In the formula of finding the perplexity of a corpus, why is it normalized based on the total number of words? Why shouldn't be normalized based on number of sentences? If # of sentences is used for ...
0
votes
0answers
14 views

What is the difference between position embedding vs positional encoding in BERT?

This post about the Transformer introduced the concept of "Positional Encoding", while at the same time, the BERT paper mentioned "Position Embedding" as an input to BERT (e.g. in Figure 2). First, ...
0
votes
0answers
15 views

With 54 observations by several thousand variables, what statistical procedures?

I have data on 54 African nations' ICT policy documents. I extracted several thousand words that are significantly different between an exemplary group of nations and the remainder. Next, I created ...
0
votes
0answers
15 views

Using perplexity to evaluate n gram model

I have built a simple n gram model for predicting the next word in a sentenced based on a maximum likelihood estimation. Currently I am using the Katz backoff algorithm to decide between n gram ...

1
2 3 4 5
19