Natural Language Processing is a set of techniques from linguistics, artificial intelligence, machine learning and statistics that aim at processing and understanding human languages.

learn more… | top users | synonyms (1)

0
votes
0answers
4 views

Varying outputs for the same train and test data

I have the below sample code in which I am using the sklearn(scikit-learn=0.16.1) 20newsgroup dataset: ...
0
votes
0answers
16 views

How to do sentence boundary detection? [on hold]

I have a string as My name is sushil feeling bad My age is 30 and hobbies are none thanks for asking me name varun i am good at cricket and my age is 40 i have done a lot of work in research area , ...
0
votes
1answer
7 views

A measure for sustained activity (dispersion)

I need a numeric measure that determines a level of sustained activity. Let me describe a situation in which this could be useful. Imagine that you have a Twitter feed in which you are tracking words ...
0
votes
0answers
14 views

Determining intentions on synonym phrases

I am building an app that works with the Stanford NLP Parser in order to annotate all the parts of the sentence. I can use those chunks to (more or less) understand what the user wants. Now, my ...
0
votes
0answers
10 views

Phenomena of undersampling the dataset

I have a question regarding undersampling the dataset. Under-sampling it's well known technique when you remove instances of the over-represented classes from the dataset until every class has equal ...
0
votes
0answers
13 views

Statistical test to determine the difference or similarity between two variables?

I want to know if there is a way to determine the probability of two variables are referring to the same thing, aka to know if there is an association between the two variables. Let’s say there is a ...
0
votes
0answers
12 views

Training word2vec embeddings - Benefits of re-using identical sentences or use only unique sentences?

I stumbled upon a thought that I am quite unsure about as I have yet to dig into the details of the word2vec algorithms. I have a rather limited-sized corpus of in-domain data containing sentences ...
2
votes
2answers
102 views

Apply word embeddings to entire document, to get a feature vector

How do I use a word embedding to map a document to a feature vector, suitable for use with supervised learning? A word embedding maps each word $w$ to a vector $v \in \mathbb{R}^d$, where $d$ is some ...
1
vote
0answers
9 views

Using Conditional random field for many valued labels

I want to use the CRF for labeling a corpus of annotated text. Each word in the corpus has its own set of labels. More specifically, the labels are the pronunciations of each word: some words like "...
0
votes
0answers
9 views

Validating how to treat unseen NGrams for Kneser Ney smoothing

I'm implementing the Kneser Ney smoothing algorithm and I would like to validate my understanding on how to treat unseen ngrams in the formula. The formula for Kneser Ney is as follows: $P_{(KN)}(...
1
vote
0answers
15 views

How to Use LSA Create Topics?

Just want to know the general process of creating document topics via LSA. For creating document clusters, I know first I should get SVD dimensions and then use k-means clustering on these SVD ...
1
vote
0answers
20 views

Post Address Extraction from webpages using Machine Learning [closed]

We are trying to extract addresses from webpages (specifically small business sites like restaurants, salons etc). I believe address is a structure (street, city, Zip.) There might be some exceptions ...
0
votes
0answers
36 views

Estimating data set size for pattern extraction

I have a dependency treebank comprised of 100 structures, which is divided into a training set and a test set. I extract some rules ((DS,PS) pairs) to convert the treebank to phrase structures. When I ...
2
votes
1answer
36 views

In Kneser Ney smoothing, how to implement the recursion in the formula?

I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams: $P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) - \...
0
votes
0answers
26 views

How to pre-process data & tune hyperparameters of Doc2Vec?

I am using doc2vec for getting document similarity (unsupervised learning). I read that we need to shuffle the input matrix to doc2vec & reduce the learning rate for better performance. But the ...
3
votes
0answers
17 views

Similarity measures and document length

I have an application where I need to measure the similarity between the (TF-IDF?) representation of two documents: $\mathbf{a}$ and $\mathbf{b}$ while still taking the document length into account. ...
0
votes
0answers
15 views

How to processing a document to extract the description of certain properties of a reference domain?

I should analyze a text in order to identify the description of certain properties of some objects belonging to a given reference domain. The objects and their properties are known, as well as the ...
1
vote
2answers
150 views

conditional probability results in value greater than 1

I am not an expert in probability theory so please bare with me. Suppose I have a one sentence corpus as follows: How to go about it It's quite obvious to see that the corpus has 5 words. Here is how ...
0
votes
0answers
30 views

Why do most natural language processing papers fail to give any significance test when reporting results?

Most natural language processing papers (> 90% from my experience) fail to give any significance test when reporting results. Is there any reason for that aside from the fact that it takes a bit of ...
1
vote
0answers
22 views

Machine Translation: WMT'15 testset, the .sgm format [closed]

The test (and development) set used in the popular wmt'15 dataset is in the format .sgm - Standard Generalized Markup Language. The file format does not seem very ...
1
vote
1answer
84 views

Different models with gensim Word2Vec on python

I am trying to apply the word2vec model implemented in the library gensim in python. I have a list of sentences (each sentences is a list of words). For instance let us have: ...
0
votes
0answers
24 views

Normalized term frequency comparisons across documents of differing length & language

I aim to infer on the prevalence of terms across and within corpora of different languages (where document length varies within and across corpora). Given Zipf’s and Heap’s laws a simple tf/n seems ...
1
vote
0answers
38 views

Machine Learning using NLP results

I have a large data set with over a million products. The NLP results look like this: A random row (reshaped) looks like this: The dataframe (image) contains information derived from the ...
3
votes
1answer
107 views

Intepreting Doc2Vec, Cosine Similarity between Doc Vectors and Word Vectors

After training a doc2vec network can you only compare word vectors with each other and doc vectors with each other? Or does it also make sense to compare word vectors with doc vectors? Well, of course ...
0
votes
0answers
24 views

Advantage of character based language models over word based

Is there an intuition why character based models language bases models are preferred over word based. For example Karpathy builds his language model by predicting the next character in Karpathy Blog. ...
0
votes
0answers
15 views

Understanding results from precision, recall, f-measure, and jaccard's distance

So I am working on a NLP problem, and I am having trouble interpreting results from my Python program. I have read a bunch of documents and extracted a bunch of terms and words which I think are ...
4
votes
1answer
45 views

Interpretation of hidden states in HMM in the part-of-speech tagging task

Let me begin with a part-of-speech tagging task. The ultimate goal: given a sentence, what is the most probable part-of-speech tag for each word in the sentence? We want to answer this question by ...
0
votes
0answers
37 views

Word2vec Vector Quality vs Number of Training Iterations

I was looking at this paper http://arxiv.org/pdf/1605.07891v1.pdf and at one point it states ...
3
votes
1answer
108 views

Difference between Log Entropy Model and TF-IDF Model?

I would like to understand what are the differences/advantages in using TF-IDF or the Log Entropy model for represeting documents and queries in an information retrieval system using diferent weights. ...
0
votes
2answers
28 views

k means clustering for larger text fields

I'm a beginner in data science/machine learning and am attempting to work through some problems on my own I am running a K-means clustering on a dataset consisting of "mission statements". These can ...
0
votes
0answers
53 views

log bilinear language models architecture

I am trying to understand this paper. The paper basically introduces a simple variation of feed forward neural network with $h$ hidden units, in which inputs are a sequence of words $(w_1, ...w_{n-1})$...
0
votes
1answer
249 views

Does the skipgram language model try to predict all context words at the same time?

In the skipgram language model (Mikolov et al., 2013), a neural network with one hidden layer tries to predict surrounding words from current words of the corpus. After training, the hidden activation ...
0
votes
2answers
76 views

Kneser-Ney for unigrams?

I was wondering if it is at all possible to use Kneser-Ney to smooth word unigram probabilites? The basic idea behind back-off is to use (n-1)-gram frequencies when an n-gram has 0 count. This is ...
0
votes
0answers
25 views

Linear model coefficients with text mining

Suppose that I have a collection of reviews about food - perhaps reviews for a restaurant or something. Also suppose that I'm interested in predicting the score from these reviews. One approach that I ...
0
votes
0answers
6 views

Transition/emission probabilities POS tagger English

is there an freely available online version of transition and emission probabilities for an HMM model used for POS tagging English text? It seems like there are many powerful existing taggers out ...
1
vote
0answers
55 views

Pros and Cons: LDA vs Neural Networks

LDA is an older approach for word representations, there are newer methods now like CBOW and Skip-gram. But what are the improvements of these models? Do they improve in every way or does LDA still ...
0
votes
0answers
39 views

How to detect the events using NLP and Machine learning?

I have text describing about events such as birth , new job , wedding , death etc .. or no event . How do i detect these events ? My approach is to form set of words and search them in text ...
0
votes
0answers
21 views

How large should be negative train set in text classification of rare category

I'm working on the classification of medical texts in order to find texts about the quite rare disease from the big set of all medical articles. I have the set of positive examples and some negative ...
0
votes
0answers
62 views

Kneser Ney smoothing, why the maths allows division by 0?

I'm studying Natural Language Processing and the various smoothing approaches. I'm finding a little hard to understand how to handle unknown words with the Kneser-Ney smoothing. In particular I'm ...
0
votes
0answers
29 views

Evaluating kmeans clustering with silhouette coefficient, weird results

I'm performing a kmeans clustering on a 22.000 documents datasets. Not knowing how many clusters I should get, I ran different k values and try to assess the validity of the clusters by determining ...
1
vote
0answers
23 views

How to choose an optimal dimension reduction factor in LSA processing

I'm performing a K-Means clustering on a 400.000 text dataset. After eliminating useless chars and removing stopwords, I get a dictionnary size of around 42000 words. So before doing the clustering, ...
2
votes
0answers
28 views

Overview research on ambiguity of words

As far as I know vector representations of word embeddings do not account for ambiguity. A single word can have different meanings f.e. "hot" can mean very good looking or very warm. Another example ...
4
votes
1answer
59 views

Do word vectors obtained via word embedding techniques really form a vector space?

Word embedding refers to feature learning techniques in natural language processing where words are mapped to vectors of real numbers in a low-dimensional space, the embedding space. Similar to ...
1
vote
1answer
94 views

What is the perplexity of a mini-language of numbers [0-9] where 0 has prob 10 times the other numbers?

I'm reading Speech and Language Processing, Jurafsky and Martin, in particular chapter 4 where they introduce perplexity see https://web.stanford.edu/~jurafsky/slp3/4.pdf (page 8-9) Here a brief ...
0
votes
0answers
18 views

Create features from a document

I have been given an assignment related to NLP and I am a newbie in this field. Train a named entity recognition system that treats the documents as strings of mentions, x . A labelling of the ...
0
votes
0answers
9 views

trainset and testset with different distributions

I am looking for the research/papers that shows that more balanced train leads to a better macro performance on highly unbalanced test. For example, there is multi-label problem, where 80% of the ...
1
vote
1answer
36 views

How may I convert Perplexity to F Measure

In the practice of Machine Learning accuracy of some models are determined by perplexity, (like LDA), while many of them (Naive Bayes, HMM,etc..) by F Measure. I like to evaluate all the models with ...
0
votes
1answer
35 views

Is there any advantage of using MEMM instead of CRF for named-entity recognition?

I wonder whether there is any advantage of using maximum-entropy Markov model (MEMM), a.k.a. conditional Markov model (CMM) instead of using conditional random fields (CRF) for named-entity ...
1
vote
0answers
97 views

Katz Backoff help calculating alpha

$Pkatz(z|x,y) =$ $P'(z|x,y), if C(x,y,z) > 0$ $α(x,y)Pkatz(z|y), else if C(x,y) > 0$ $P'(z), otherwise.$ $Pkatz(z|y) =$ $P'(z|y),ifC(y,z)>0$ $α(y)P' (z), otherwise.$ $$P'(...
2
votes
0answers
43 views

Alpha on Katz Backoff using Simple Good-Turing

I'm building an n-gram language model to predict the next word, I've implemented a simple Good-Turing smoothing on all my probabilities and have calculated the P0(mass probability of unseen event). I ...