The natural-language tag has no wiki summary.
3
votes
1answer
59 views
Intutive difference between hidden Markov models and conditional random fields
I understand that HMM are generative models, and CRF are discriminative models. I also understand how CRFs' are designed and used. What I do not understand is how they are different from HMMs'? I read ...
3
votes
0answers
40 views
Latent Semantic Analysis - Co-occurrence of words
Let $A[n\times m]$ represents the term-document matrix, where, $n$ is the number of terms and $m$ is the number of documents. This matrix can be composed into 3 matrices (SVD decomposition) such as,
...
0
votes
0answers
28 views
Understanding the derivation of an equation in LDA modeling
When reading the derivation of LDA models, I usually get the following equations.
I do not quite understand the second step, where $p(\mathbf{z}_{-i},\mathbf{w}|\alpha,\beta)$ was removed. Is that ...
1
vote
1answer
98 views
Loop over Tokens in RapidMiner's Text Processing Plugin
is there any possibility to iterate over the tokens of a text document within RapidMiner? My first try was to window the document after tokenisation. But this seems very complicated.
I'm doing this ...
2
votes
1answer
36 views
How to extract ngrams from ambigous text after lemmatization?
After lemmatization of text I have a sequence of sets of lemmas, because every word can correspond to more than one lemma. How should I extract ngram statistics based on that? The only thing that ...
0
votes
0answers
49 views
Are my HMM calculations going fine? [closed]
I was trying to understand the hidden Markov model (HMM) and to do some calculations, and I got some doubts. I attach my study in this Google Drive File.
Can you check if my calculations are fine?
I ...
1
vote
1answer
50 views
Independence assumption in maximum entropy models in NLP
I am reading Klein and Manning's notes on Maximum Entropy for Natural Language Processing. On slide 22, they have an equation saying,
$P(C|D,\lambda) = \Pi _{(c,d)\in (C,D)} P(c|d,\lambda)$. I am not ...
2
votes
0answers
42 views
Best method for estimating divergence with unequal sample sizes?
Background
I have several datasets of word frequencies where some datasets have much more data than others: from 500 samples to 3000 samples. I also have large reference corpora with millions of ...
0
votes
1answer
92 views
Regarding the feature generation method with SVM-based classification method
When using SVM to build classifier for a collection of documents, we can use term occurrence, term frequency or even TF/IDF. I would like to know what are the main disadvantages of using term ...
2
votes
0answers
17 views
what is the way of dealing with textual valued feature vectors for classification task?
I aim to work on twitter data for sentiment analysis but I am curios for the way of dealing such a huge number of textual features (words). Is using the Bag-Of-Words approach is the best? However I've ...
-1
votes
1answer
50 views
verb/predicate detection/flagging in sentence [closed]
What would be good algorithms to detect verbs/predicate in a sentence?
With links if possible.
What software uses those methods?
1
vote
0answers
51 views
Using sentiment lexicons or all words processing for sentiment analysis?
I am learning sentiment analysis to apply it to twitter real time data to predict user's mood. I ponder about using which alternative way to do that data mining job.
Use all words to process and ...
3
votes
1answer
124 views
Latent Dirichlet allocation Implementation
I'm looking for some LDA implementation. I know about this one, MALLET but it is coded in Java and I need some more performant.
Can someone give me some reference?
5
votes
1answer
290 views
Why is tf-idf used in conjunction with SVMs for classifying documents?
Term frequency - inverse document frequency is term count within a document weighted against the term's ubiquity within the corpus. This weight is based on the principle that terms occurring in ...
2
votes
0answers
65 views
Using n-grams to find data that does not 'belong'
Recently I posted a question over in CS.SE dealing with methods of classifying data. Essentially the problem is that I have a collection of strings (100's of thousands). Most of these strings are ...
2
votes
1answer
49 views
How to update a dynamic language model dataset?
I'm a statistics novice and I need help with a natural language problem.
I'm writing a word-prediction algorithm for a mobile app. I'm using a unigram language model of word/count pairs where count ...
6
votes
2answers
479 views
Topic models and word co-occurrence methods
Popular topic models like LDA usually cluster words that tend to co-occur together into the same topic (cluster).
What is the main difference between such topic models, and other simple ...
4
votes
0answers
64 views
Language modeling: why is adding up to 1 so important?
(if this venue is inappropriate, feel free to migrate it)
In many natural language processing applications such as spelling correction, machine translation and speech recognition, we use language ...
0
votes
1answer
67 views
Lucene-based text feature construction
When doing the feature construction for text mining, does Lucene has a better performance in terms of classification/clustering result than the traditional bag-of-word approach?
2
votes
1answer
297 views
Different size of vocabulary made by Weka and R's tm
I own around 40,000 text files for preprocessing (in purpose of document classification). I used R (with tm package) for prototype and now looking for a equivalent Java library for products.
...
1
vote
4answers
557 views
Software or libraries to create doc-term matrix
does anyone know some Java libraries to create the document-term matrix for a large number (50,000) of documents ? I wish this library encompasses preprocessing functionalities, like stop-word and ...
2
votes
0answers
81 views
Feature construction for text mining
In the text mining, besides N-gram model, what are the state-of-art models for building feature space while capturing the dependence among the different words, or capturing the semantic meaning in the ...
1
vote
1answer
65 views
Larger ngrams vs nested ngrams?
Whenever I see people using ngrams - I often see them looking for higher chains like 4-grams, 5-grams and so on. However, I'm wondering why I never see any mention of "nested ngrams" (I'm not sure ...
3
votes
1answer
276 views
Regarding using bigram (N-gram) model to build feature vector for text document
A traditional approach of feature construction for text mining is bag-of-words approach, and can be enhanced using tf-idf for setting up the feature vector characterizing a given text document. At ...
2
votes
1answer
157 views
Software packages that can construct feature representation for a given text file using N-gram model
Are there any open source software packages, (Java, Matlab, R) that can generate a feature representation for a text document using N-gram model?
1
vote
1answer
117 views
Regarding the R packages that share the similar functionalities of NLTK toolkit
Are there any R packages that share the similar functionalities of NLTK toolkit?
1
vote
0answers
29 views
The approach of labelling a collection of documents using a shared topic
I have a collection of documents, and know they may share a single topic. Is there a way to identify this topic? I know LDA (Latent Dirichlet Allocation) is an approach. But LDA result is to associate ...
2
votes
0answers
62 views
Statistical model of a website
I know that HMMs can be used to construct statistical models of text. Thus, we can generate text according to this model, and compute the likelihood of a text sample under the model.
What tools are ...
4
votes
1answer
185 views
At what n do n-grams become counterproductive?
When doing natural language processing, one can take a corpus and evaluate the probability of the next word occurring in a sequence of n. n is usually chosen as 2 or 3 (bigrams and trigrams).
Is ...
14
votes
2answers
245 views
Why does Natural Language Processing not fall under Machine Learning domain? [closed]
I encounter it in many books as well as web. Natural Language Processing and Machine Learning are said to be different subsets of Artificial Intelligence. Why is it? We can achieve results of Natural ...
5
votes
1answer
503 views
TF-IDF cutoff percentage for tweets
I'm currently trying to analyze Tweets and classify them as either positive, negative, or neutral using the NLTK library in Python.
I can see that there's potential in the approach that I'm taking, ...
3
votes
1answer
346 views
Natural language processing application in algorithmic trading and sports betting
I'm want to do a PhD in NLP and I'm defining the topic at the moment.
I've heard that NLP can be used to track online sentiment which in turn can be used for algorithmic trading on stock markets. See ...
1
vote
1answer
54 views
I am looking for a methodology for categorizing keywords
I.e. I want to put in Baseball and get out Sports & Outdoors.
2
votes
0answers
160 views
Log-likelihood ratio in document summarization
I initially asked this on stack overflow and was referred to this site, so here goes:
I am implementing some unsupervised methods of content-selection/extraction based document summarization and I'm ...
2
votes
0answers
63 views
Handling NLP probability results
I'm a programmer without a statistics background. I've been working with NLP lately to classify documents, and I'm pretty up to speed with NLP concepts. I've gotten to the point where the NLP ...
1
vote
0answers
57 views
How to characterize a problem of standardizing product descriptions
I'm looking for some advice for where to start on this problem. Let's say I have sales transaction data from a number of different retailers that all sell the same products. Even though they are ...
1
vote
1answer
399 views
In natural language parsing, what is the feature function?
I'm working in the area of natural language processing, to be specific I'm reviewing the parsers that take advantage of data-mining techniques.
I've read an introduction to natural language ...
