Natural Language Processing is a set of techniques from linguistics, artificial intelligence, machine learning and statistics that aim at processing and understanding human languages.

learn more… | top users | synonyms (1)

0
votes
0answers
6 views

Validating how to treat unseen NGrams for Kneser Ney smoothing

I'm implementing the Kneser Ney smoothing algorithm and I would like to validate my understanding on how to treat unseen ngrams in the formula. The formula for Kneser Ney is as follows: $P_{(KN)}(...
1
vote
0answers
12 views

How to Use LSA Create Topics?

Just want to know the general process of creating document topics via LSA. For creating document clusters, I know first I should get SVD dimensions and then use k-means clustering on these SVD ...
1
vote
0answers
15 views

Post Address Extraction from webpages using Machine Learning [on hold]

We are trying to extract addresses from webpages (specifically small business sites like restaurants, salons etc). I believe address is a structure (street, city, Zip.) There might be some exceptions ...
0
votes
0answers
22 views
+50

Estimating data set size for pattern extraction

I have a dependency treebank comprised of 100 structures, which is divided into a training set and a test set. I extract some rules ((DS,PS) pairs) to convert the treebank to phrase structures. When I ...
2
votes
1answer
28 views

In Kneser Ney smoothing, how to implement the recursion in the formula?

I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams: $P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) - \...
0
votes
0answers
15 views

How to pre-process data & tune hyperparameters of Doc2Vec?

I am using doc2vec for getting document similarity (unsupervised learning). I read that we need to shuffle the input matrix to doc2vec & reduce the learning rate for better performance. But the ...
3
votes
0answers
16 views

Similarity measures and document length

I have an application where I need to measure the similarity between the (TF-IDF?) representation of two documents: $\mathbf{a}$ and $\mathbf{b}$ while still taking the document length into account. ...
0
votes
0answers
25 views

Building AI chat bot [closed]

I am trying to build a conversational chatbot using deeplearning . I am thinking to use tv subtitles corpus . But i am not getting how to use these subtitles as train and test sets . Using directly ...
0
votes
0answers
15 views

How to processing a document to extract the description of certain properties of a reference domain?

I should analyze a text in order to identify the description of certain properties of some objects belonging to a given reference domain. The objects and their properties are known, as well as the ...
1
vote
2answers
148 views

conditional probability results in value greater than 1

I am not an expert in probability theory so please bare with me. Suppose I have a one sentence corpus as follows: How to go about it It's quite obvious to see that the corpus has 5 words. Here is how ...
0
votes
0answers
29 views

Why do most natural language processing papers fail to give any significance test when reporting results?

Most natural language processing papers (> 90% from my experience) fail to give any significance test when reporting results. Is there any reason for that aside from the fact that it takes a bit of ...
1
vote
0answers
21 views

Machine Translation: WMT'15 testset, the .sgm format [closed]

The test (and development) set used in the popular wmt'15 dataset is in the format .sgm - Standard Generalized Markup Language. The file format does not seem very ...
1
vote
1answer
50 views

Different models with gensim Word2Vec on python

I am trying to apply the word2vec model implemented in the library gensim in python. I have a list of sentences (each sentences is a list of words). For instance let us have: ...
0
votes
0answers
23 views

Normalized term frequency comparisons across documents of differing length & language

I aim to infer on the prevalence of terms across and within corpora of different languages (where document length varies within and across corpora). Given Zipf’s and Heap’s laws a simple tf/n seems ...
1
vote
0answers
37 views

Machine Learning using NLP results

I have a large data set with over a million products. The NLP results look like this: A random row (reshaped) looks like this: The dataframe (image) contains information derived from the ...
3
votes
1answer
82 views

Intepreting Doc2Vec, Cosine Similarity between Doc Vectors and Word Vectors

After training a doc2vec network can you only compare word vectors with each other and doc vectors with each other? Or does it also make sense to compare word vectors with doc vectors? Well, of course ...
0
votes
0answers
18 views

Advantage of character based language models over word based

Is there an intuition why character based models language bases models are preferred over word based. For example Karpathy builds his language model by predicting the next character in Karpathy Blog. ...
0
votes
0answers
10 views

Understanding results from precision, recall, f-measure, and jaccard's distance

So I am working on a NLP problem, and I am having trouble interpreting results from my Python program. I have read a bunch of documents and extracted a bunch of terms and words which I think are ...
3
votes
1answer
35 views

Interpretation of hidden states in HMM in the part-of-speech tagging task

Let me begin with a part-of-speech tagging task. The ultimate goal: given a sentence, what is the most probable part-of-speech tag for each word in the sentence? We want to answer this question by ...
0
votes
0answers
24 views

Word2vec Vector Quality vs Number of Training Iterations

I was looking at this paper http://arxiv.org/pdf/1605.07891v1.pdf and at one point it states ...
3
votes
1answer
73 views

Difference between Log Entropy Model and TF-IDF Model?

I would like to understand what are the differences/advantages in using TF-IDF or the Log Entropy model for represeting documents and queries in an information retrieval system using diferent weights. ...
0
votes
2answers
27 views

k means clustering for larger text fields

I'm a beginner in data science/machine learning and am attempting to work through some problems on my own I am running a K-means clustering on a dataset consisting of "mission statements". These can ...
0
votes
0answers
49 views

log bilinear language models architecture

I am trying to understand this paper. The paper basically introduces a simple variation of feed forward neural network with $h$ hidden units, in which inputs are a sequence of words $(w_1, ...w_{n-1})$...
0
votes
1answer
245 views

Does the skipgram language model try to predict all context words at the same time?

In the skipgram language model (Mikolov et al., 2013), a neural network with one hidden layer tries to predict surrounding words from current words of the corpus. After training, the hidden activation ...
0
votes
2answers
69 views

Kneser-Ney for unigrams?

I was wondering if it is at all possible to use Kneser-Ney to smooth word unigram probabilites? The basic idea behind back-off is to use (n-1)-gram frequencies when an n-gram has 0 count. This is ...
0
votes
0answers
25 views

Linear model coefficients with text mining

Suppose that I have a collection of reviews about food - perhaps reviews for a restaurant or something. Also suppose that I'm interested in predicting the score from these reviews. One approach that I ...
0
votes
0answers
6 views

Transition/emission probabilities POS tagger English

is there an freely available online version of transition and emission probabilities for an HMM model used for POS tagging English text? It seems like there are many powerful existing taggers out ...
1
vote
0answers
48 views

Pros and Cons: LDA vs Neural Networks

LDA is an older approach for word representations, there are newer methods now like CBOW and Skip-gram. But what are the improvements of these models? Do they improve in every way or does LDA still ...
0
votes
0answers
34 views

How to detect the events using NLP and Machine learning?

I have text describing about events such as birth , new job , wedding , death etc .. or no event . How do i detect these events ? My approach is to form set of words and search them in text ...
0
votes
0answers
21 views

How large should be negative train set in text classification of rare category

I'm working on the classification of medical texts in order to find texts about the quite rare disease from the big set of all medical articles. I have the set of positive examples and some negative ...
0
votes
0answers
57 views

Kneser Ney smoothing, why the maths allows division by 0?

I'm studying Natural Language Processing and the various smoothing approaches. I'm finding a little hard to understand how to handle unknown words with the Kneser-Ney smoothing. In particular I'm ...
0
votes
0answers
25 views

Evaluating kmeans clustering with silhouette coefficient, weird results

I'm performing a kmeans clustering on a 22.000 documents datasets. Not knowing how many clusters I should get, I ran different k values and try to assess the validity of the clusters by determining ...
1
vote
0answers
17 views

How to choose an optimal dimension reduction factor in LSA processing

I'm performing a K-Means clustering on a 400.000 text dataset. After eliminating useless chars and removing stopwords, I get a dictionnary size of around 42000 words. So before doing the clustering, ...
2
votes
0answers
23 views

Overview research on ambiguity of words

As far as I know vector representations of word embeddings do not account for ambiguity. A single word can have different meanings f.e. "hot" can mean very good looking or very warm. Another example ...
4
votes
1answer
49 views

Do word vectors obtained via word embedding techniques really form a vector space?

Word embedding refers to feature learning techniques in natural language processing where words are mapped to vectors of real numbers in a low-dimensional space, the embedding space. Similar to ...
1
vote
1answer
92 views

What is the perplexity of a mini-language of numbers [0-9] where 0 has prob 10 times the other numbers?

I'm reading Speech and Language Processing, Jurafsky and Martin, in particular chapter 4 where they introduce perplexity see https://web.stanford.edu/~jurafsky/slp3/4.pdf (page 8-9) Here a brief ...
0
votes
0answers
18 views

Create features from a document

I have been given an assignment related to NLP and I am a newbie in this field. Train a named entity recognition system that treats the documents as strings of mentions, x . A labelling of the ...
0
votes
0answers
9 views

trainset and testset with different distributions

I am looking for the research/papers that shows that more balanced train leads to a better macro performance on highly unbalanced test. For example, there is multi-label problem, where 80% of the ...
1
vote
1answer
28 views

How may I convert Perplexity to F Measure

In the practice of Machine Learning accuracy of some models are determined by perplexity, (like LDA), while many of them (Naive Bayes, HMM,etc..) by F Measure. I like to evaluate all the models with ...
0
votes
1answer
26 views

Is there any advantage of using MEMM instead of CRF for named-entity recognition?

I wonder whether there is any advantage of using maximum-entropy Markov model (MEMM), a.k.a. conditional Markov model (CMM) instead of using conditional random fields (CRF) for named-entity ...
1
vote
0answers
87 views

Katz Backoff help calculating alpha

$Pkatz(z|x,y) =$ $P'(z|x,y), if C(x,y,z) > 0$ $α(x,y)Pkatz(z|y), else if C(x,y) > 0$ $P'(z), otherwise.$ $Pkatz(z|y) =$ $P'(z|y),ifC(y,z)>0$ $α(y)P' (z), otherwise.$ $$P'(...
2
votes
0answers
37 views

Alpha on Katz Backoff using Simple Good-Turing

I'm building an n-gram language model to predict the next word, I've implemented a simple Good-Turing smoothing on all my probabilities and have calculated the P0(mass probability of unseen event). I ...
1
vote
0answers
18 views

Is there any package in R for Sound like analysys of text [closed]

The words "Jhon" and "Joan" may sound similar although spelling is different. Is there any package for "Sound like Analysis" in R
0
votes
1answer
31 views

Doubt about feature selection

I'm working on a text classification problem using Python and NLTK. I've got two frequency distributions, one for each class (it's basically a binary classification). So, my doubt it's if there's a ...
1
vote
0answers
34 views

Simple Good-Turing Probabilities higher than old probabilities

I've implemented Simple Good-Turing to get new probabilities of unigrams on my Corpus. Everything works fine. I'm just confused on how come the probability of a word after the Good-Turing discount be ...
0
votes
0answers
21 views

nltk: odd outputs from bleu_score

For machine translation purposes I use bleu score, which seems to be the validation mechanism of choice (used in the sutskever 2014 sequence-to-sequence). The purpose is to get as high bleu as ...
1
vote
0answers
33 views

How to create a realistic multi-label data set of web pages? [closed]

I am working on a project which is a web filter. The web filter is supposed to classify web pages using any multi-label classification technique and then block them if anyone of the category is found ...
2
votes
1answer
126 views

Handling unknown words in language modeling tasks using LSTM

For a natural language processing (NLP) task one often uses word2vec vectors as an embedding for the words. However, there may be many unknown words that are not captured by the word2vec vectors ...
1
vote
0answers
354 views

Text classification with word2vec and neural nets [spacy.io, keras]

I have about 300.000 messages with bodies and titles at hand. ~20% are of them labeled positive. Right now, I run the word2vec feature generation with spacy.io (excellent library btw.), generating ...
0
votes
0answers
15 views

Stacked CRF implementation

I am looking for a library that can train a stacked conditional random field (CRF). I plan to use it for natural language processing purposes. Ideally, Python interface, works on Linux, and multi-...