Questions tagged [natural-language]

Natural Language Processing is a set of techniques from linguistics, artificial intelligence, machine learning and statistics that aim at processing and understanding human languages.

0
votes
0answers
4 views

Metric for ranked keyword identification

I am trying to determine which metric(s) to use to evaluate the "coverage" of a lexicon (list of words) with respect to a ranked list of significant keywords I have extracted from two different ...
0
votes
0answers
9 views

Denoising autoencoders vs masked approaches

I am not an expert in language modelling domain. There are mainly two approaches that are being used nowadays. Denoising autoencoders and ...
0
votes
0answers
34 views

LSTM model for text generation in Keras [on hold]

I am new to Keras and currently following these two tutorials to generate new name variants. The input data is a name and the target variable is called variant which represents a different way the ...
0
votes
0answers
9 views

Accuracy of fine-tuning BERT varied significantly based on epochs for intent classification task [on hold]

I used Bert base uncased as embedding and doing simple cosine similarity for intent classification in my dataset (around ...
0
votes
0answers
6 views

How to interpret the transition and feature weights in a Conditional Random Field model?

Conditional Random Fields model have been a popular method for Named Entity Recognition as it accounts for statistical dependencies between entities and can include observed features that can aid with ...
1
vote
1answer
69 views

Is it true that Bahdanau's attention mechanism is not Global like Luong's?

I was reading the pytorch tutorial on a chatbot task and attention where it said: Luong et al. improved upon Bahdanau et al.’s groundwork by creating “Global attention”. The key difference is that ...
2
votes
0answers
11 views

Optimal compounding rate for minibatches SpaCy NER model

I am trying to train a blank NER model using SpaCy. I have 500000 samples of text data and want to train 9 entities. This is the configuration for the mini-batches I am using: ...
0
votes
0answers
6 views

How was the perplexity of the Brown corpus measured?

I used the following code to calculate the perplexity of a corpus but it is giving me extremely low answer for bigram analyses. For Brown Corpus I am getting a result of 4.6. ...
2
votes
1answer
13 views

Is it possible to take a pertained word embedding, trained on a general vocabulary and make it domain specific?

Suppose that I have an NLP task that I want to keep restricted to the vocabulary of a specific domain. This vocabulary is a subset of a language as a whole, but still presents too large of a corpus ...
1
vote
1answer
35 views

Why are integers not used for vocabularies in Natural Language Processing (NLP)?

I know that this might sound like a really dum or naive question but I believe it's not (I hope). I've noticed that a default used to be to have one hot vectors to encode words in a vocabulary. But ...
0
votes
1answer
36 views

gensim LdaModel - How to reduce the number of words in each topic?

I'm trying to get more sparse topics (Less overlaps between different topics). https://radimrehurek.com/gensim/models/ldamodel.html I know it should be determined by the alpha parameter. I've ...
0
votes
0answers
7 views

Is negative Viterbi Loss possible?

So I'm training a sequence-labeling model with a BiLSTM-CRF architecture, and I am getting negative values on Viterbi Loss. Is this possible? I'm using the following formula in my code, as specified ...
0
votes
0answers
8 views

Metrics used in evaluation of topic models

I know that perplexity requires a held out corpus. I am new to coherence measures. What are intrinsic and extrinsic coherence measures? Which one of them require an external corpus like Wikipedia ...
0
votes
0answers
7 views

Evaluating topics generated by topic models

I am working on topic modelling of Amazon reviews and descriptions. I read that the coherence measure is better because it is correlated with human topic interpretation. The online resources explain ...
0
votes
1answer
39 views

Current SOTA techniques to handle morphological variations in words (e.g. plurals, verb conjugations, hyphens, etc.) in NLP?

I need to process natural language sentences in which words can appear with morphological variations: car -> cars; play -> playing, played; etc. There might be hyphens also, e.g. "dog-friendly hotel", ...
0
votes
0answers
14 views

Treating Word Embeddings as Samples From Random Variables

Suppose I want to specify some probabilistic clustering model (such as a mixture model or lda) over words, and instead of using the traditional method of representing words as an indicator vector $z$, ...
1
vote
1answer
26 views

Checking for plagiarism with a proportion test

I'm reviewing research proposals. The proposal is, in human eyes, practically the same as a 2017 degree thesis. The thesis guide professor is the principal investigator of the current project. I used ...
0
votes
0answers
4 views

Can I use two differently annotated corpora for training and evaluation of a NER system?

I have two text corpora's in English. Corpora 1: One of them is specifically annotated (with labels A and B) by third-party to act as training data for NER systems (for example, bidirectional LSTMs). ...
3
votes
2answers
135 views

Why do attention models need to choose a maximum sentence length?

I was going through the seq2seq-translation tutorial on pytorch and found the following sentence: Because there are sentences of all sizes in the training data, to actually create and train this ...
2
votes
1answer
33 views

Balanced datasets are almost all predicted negative

Problem I am trying to do sentiment analysis using pretrained word vectors GloVe, which is essentially a look-up table that maps word to a fix-dimension vector. Since GloVe is initially designed to ...
0
votes
0answers
21 views

Classify documents using a set of known vocabularies

I have a bunch of documents that I want to classify which ones talk about soccer (unsupervised learning, I do not want to manually label the documents). One way I am thinking about is to go online ...
0
votes
0answers
20 views

Why can we approximate the joint probability distribution using the output vector of an LSTM?

In the paper, Contextual String Embeddings for Sequence Labeling, the authors state that \begin{equation} P(x_{0:T}) = \prod_{t=0}^T P(x_t|x_{0:t-1}) \end{equation} They also state that, in the LSTM ...
0
votes
0answers
4 views

Can we achieve higher accuracy for a PFCG parser by changing POS TAGs?

I'm given the following question. Can we achieve higher accuracy for a PFCG parser by changing POS TAGs? I guess we can change the granularity of POS tags and achieve higher accuracy, when we have ...
0
votes
1answer
16 views

Transformer based decoding

Can the decoder in a transformer model be parallelized like the encoder? As far as I understand the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder ...
0
votes
1answer
21 views

What is the intuition behind the positional cosine encoding in the transformer network?

I don't understand how adding the cosine encodings/functions to each of the dimension of the word vector embedding enables the network to "understand" where each word is situated in the sentence. ...
0
votes
0answers
8 views

Binary text classification with a variable amount of documents per labeled datapoint

I have a dataset with a label TRUE or FALSE for each person, but each person has multiple documents associated with them (emails ...
0
votes
0answers
4 views

Why does HMM model for POS tagger works better with less data

The following is the result of using an HMM model for POS tagging a corpus. The following shows the size of training data and the precision on 1000 words test data: ...
0
votes
0answers
14 views

What does a word embedding's dimension signify?

I'm currently studying NLP and had a question regarding word embeddings. My understanding of a word embedding is that it is, simply put, a modular way of expressing words and phrases as vector ...
0
votes
0answers
9 views

How does short-term dependency improve performance for NLP models?

I was reading a paper titled Sequence to Sequence Learning with Neural Networks (Sutskevers, Vinyals, and Le - 2014 NIPS) and had a question regarding the concepts of "short-term dependency" and "long-...
0
votes
1answer
24 views

using latent dirichlet allocation to reduce the number of dimensions in bag of words model?

Does anyone have experience reducing the dimensions in a traditional bag of words model? For example, if you want to train a decision tree on a large set of reviews, the size of the vocabulary ...
0
votes
0answers
9 views

Is pooling acceptable to evaluate information extraction?

When dealing with information extraction of unbalanced classes (e.g. the desired class has a prevalence of 0.5%), the required sample size for validation might be huge (thousands of cases and more), ...
0
votes
1answer
40 views

A model (neural network) for sets of arbitrary length [closed]

I've been searching for a model that is close to RNN (is well suited for investigating sets of arbitrary lengths) but is insensitive to order. I'm aware of bidirectional RNNs. I've also found a 'bag ...
0
votes
0answers
33 views

Why does 4-gram work better than trigram or bigram or unigram in my experiments?

In a binary classification task, I used Logistic regression, decision tree and Adaboost with decision tree (max_depth=1). For each of the machine learning task, I used GridSearchCV to choose the ...
0
votes
0answers
43 views

BERT Classification fine tuning for Q & A

I want to fine-tune BERT for Q & A in a different way than the SQuAD mission: I have pairs of (question, answer) Part of them are the correct answer (Label - 1) Part of them are the incorrect ...
0
votes
0answers
25 views

How to Calculate Perplexity or cross entropy from Probability Distribution for a certain sentence?

I have a list of words starting with the letter "s" and their frequency count. From this, I'm trying to build a language model. I don't have the whole text, so I can't do the conditional probability, ...
0
votes
0answers
13 views

Find categories of question using a pre-defined dictionary

I want to create a function that will automatically suggests categories when a user input a questions. The first step that I have done is to create a pre-defined dictionary with keywords and values (...
0
votes
0answers
10 views

Interpretation of NLP pipeline for topic discovery using gaussian mixture model clustering

I built a pipeline that does the following to discover topics out of a (very big: 50k docs per ~350 terms) Term Document Matrix: Compute the TfIdf score for each Term x Document pair; Rescale each ...
0
votes
1answer
12 views

Interpretation of the following logistic regression problem

I have a function that gives the probability of Y=1 given X i.e P(Y=1|X)=f(wX). This function is dependent on variables w and X and I have to give the range of w ...
0
votes
0answers
27 views

Efficiently normalize word embeddings

I'm using glove word embedding and would like to [-1,1] normalize it using python. The data is in the format of a dict with the word as key and a ...
0
votes
0answers
22 views

Accuracy of RNN getting stuck after 90% [duplicate]

I am using Keras RNN Cell to perform parts of speech tagging. The architecture is as follows(I cannot put the code because of privacy reasons) : An embedding layer of of 40 units of shape (...
0
votes
1answer
16 views

How does one use Convolutional Neural Nets (CNNs) on varying size sentences for NLP so that the final fully connected layer can remain fixed size?

I wanted to use CNNs to classify sentences. The sentences are varying length. I am going to use standard Word Embeddings (any sort of pre-trained vectors) as features for each word then concatenate ...
0
votes
0answers
13 views

NLP - how do you randomly draw negative samples?

From my understanding, negative sampling randomly samples K negative samples from a noise distribution, P(w). The noise ...
0
votes
0answers
13 views

Pointwise Mutual Information using spacy or just detailed explaination

So, I have been trying to play around with NLP recently and decided to work on a project involving Emotional Analysis. I have been following this particular research, http://www.cse.yorku.ca/~aan/...
1
vote
1answer
30 views

Having trouble figuring out how loss was calculated for SQuAD task in BERT paper

The BERT Paper https://arxiv.org/pdf/1810.04805.pdf Section 4.2 covers the SQuAD training. So from my understanding, there are two extra parameters trained, they are two vectors with the same ...
0
votes
0answers
13 views

What kind of errors could be responsible for low loss but disastrous BLEU in neural machine translation?

Please notice: I originally asked this question in Stackoverflow but I have been asked to move it here. I'm working on a custom neural machine translation model with the Fairseq framework (PyTorch), ...
0
votes
0answers
5 views

Implementation of Traditional Language Models

Problem I am now reading this paper (A Bit of Progress in Language Modeling) to know language modeling techniques prior to methods based on neural network. However, since this paper is rather old (...
0
votes
0answers
17 views

Are these Multi-label document classification experiment steps sensible?

I plan to filter an input document using 4 different labels. Just for an example, a document discussing about movie summary needs to be labeled with 4 labels (Romance, Drama, Fiction, Hollywood). ...
0
votes
2answers
33 views

Measuring similarity between document and category

Lets say I have a word embedding model, a set of documents and N categories. Lets say the categories are "cars" and "planes". I want to categorize the documents as either being about cars or planes. ...
0
votes
0answers
6 views

What would be the biggest considerations in using an SVM for NLP?

Evaluating a linear SVM on an NLP corpus where there are 150,000 data examples but each language sample is reasonably short(10-15 words). This is evaluated against a code that is a topic. For example "...
0
votes
0answers
10 views

Term x term matrix for text clustering. What to do with subterms of n-grams

I am doing topic discovery on a large corpus. Reading here and there I found some papers saying that in case of big, sparse document x term matrices is better to create first a term x term similarity ...