The tag has no wiki summary.

learn more… | top users | synonyms

-2
votes
0answers
38 views

Technologies behind Siri and Google Now?

I'm interested to know how Siri and Google Now works. What are the technologies behind them? I don't mean voice recognition (which is obvious) but the other stuff. Like interpreting the input and ...
0
votes
0answers
10 views

Issue With class imbalances in classification problem

I am working on a word sense disambiguation problem. Specifically, I am using decision lists to classify the ambiguous word. Decision lists work in the following way. The quantity ...
0
votes
0answers
24 views

Word Sense Disambiguation in Practice

I have a question that might seem very obvious but I don't really have a good answer for it. There are many algorithms out there that deal with word sense disambiguation but all of the ones that I ...
1
vote
0answers
35 views

How to extract structured information from a text string?

I have a text string containing unstructured data and I would like to analyze it in order to extract structured information. In particular, this text string specifies when a service is operational ...
1
vote
0answers
51 views

Keyword probabilities

I'm not a huge stats buff and am wondering what the best approach to my problem is. Say I have a list of PPC* keywords and if a desired action was taken. Let's say I put this data into two word ...
0
votes
0answers
22 views

MEMM label bias problem

I study MEMM model with application to POS tagging. The question I found that the MEMM model has one drawback - "label bias problem". More formally, The transition leaving a given state compete ...
0
votes
1answer
60 views

Multiple Bernoulli and Multinomial Distirbution

It's well known that language can be modeled by Multinomial distribution and Multiple Bernoulli distribution. So far I don't see any advantage of Multiple Bernoulli distribution representation over ...
2
votes
0answers
41 views

Expectation Maximization Clarification

I found very helpful tutorial regarding EM algorithm. The example and the picture from the tutorial is simply brilliant. Related question about calculating probabilities how does expectation ...
0
votes
1answer
29 views

Improvement on duplicating instances

I have a task of Relationship extraction. There are some set of predefined relations in the corpus. I need to train classifier to recognize the type of relation or the lack of relation between every ...
1
vote
2answers
84 views

Most important journals in data mining/ML, NLP and IR?

Can you please provide with me with the names of the most important journals in data mining, machine learning, natural language processing and information retrieval?
1
vote
0answers
34 views

Calculating and Normalizing ngram relevancy scores from free text extraction

I currently look for a set of ngrams in many sets of documents to establish a relevancy score for each set - eg. I look for the n-gram "adhesive tape" in ~1M sets of 1-500 documents. The values I ...
0
votes
0answers
35 views

How to build a relevant text classifier?

I would like to build a message classification system which classifies a given message into either of 2 class - Relevant/Not. I don't have any labelled dataset. I only have certain keywords which ...
0
votes
0answers
100 views

Katz backoff for n-gram language models

I have defined unigram, bigram and trigram (language) models with my training data. Now I am checking their fitness towards a test data by using Katz backoff with a fixed discount of 0.5 (mo ...
0
votes
0answers
20 views

Evaluate text rarity in document set

I would like to evaluate the rarity of each sentence in a document set. Please let me know the state-of-the-art or a survey paper on this task. I've already checked several papers in the document ...
2
votes
1answer
87 views

Regression analysis in R using text field?

I'm working in R. I'd like to run a regression analysis for predicting price against terms in a text field. I have a dataset of jewellery auction listings, with price paid, date, and an unstructured ...
0
votes
0answers
17 views

Typical sequence of analyzing text

While going about examining textual data, do you have any process flow that works for you? What I do as of now: data collection importing into a corpus coding cleaning preprocessing: whitespace, ...
1
vote
0answers
16 views

Early papers using statistical models for word completion?

I am trying to find references about (early, as in pre-2000, but i would really love something pre-1990) statistical language models in word completion (like T-9, or Google's search autocomplete), but ...
3
votes
2answers
141 views

Is LSA and topic clustering easier in European languages similar to English?

I was watching a talk on latent semantic analysis and the speaker described experience applying LSA and REALLY messy data. He concluded that it demonstrated the difficulty of disambiguation of ...
0
votes
1answer
40 views

Time-delayed neural networks - Learning the “max” layer

In Unified Architecture for NLP paper time-delayed neural network proposed as a way to deal with variable length input. Input window slides over sequence and label each output with "time", then next ...
5
votes
1answer
131 views

Using text mining/natural language processing tools for econometrics

I am not sure whether this question is fully appropriate here, if not, please delete. I am a grad student in economics. For a project which investigates issues in social insurances, I have access to ...
2
votes
1answer
80 views

Markov chain getting stuck due to insufficient data samples

There is a lot of theory on Markov models and output generation out there, but I cannot locate any information on models getting stuck. I'm trying to create a model of a data set using a Markov ...
6
votes
1answer
516 views

Intutive difference between hidden Markov models and conditional random fields

I understand that HMM are generative models, and CRF are discriminative models. I also understand how CRFs' are designed and used. What I do not understand is how they are different from HMMs'? I read ...
3
votes
0answers
98 views

Latent Semantic Analysis - Co-occurrence of words

Let $A[n\times m]$ represents the term-document matrix, where, $n$ is the number of terms and $m$ is the number of documents. This matrix can be composed into 3 matrices (SVD decomposition) such as, ...
0
votes
0answers
98 views

Understanding the derivation of an equation in LDA modeling

When reading the derivation of LDA models, I usually get the following equations. I do not quite understand the second step, where $p(\mathbf{z}_{-i},\mathbf{w}|\alpha,\beta)$ was removed. Is that ...
1
vote
1answer
543 views

Loop over Tokens in RapidMiner's Text Processing Plugin

is there any possibility to iterate over the tokens of a text document within RapidMiner? My first try was to window the document after tokenisation. But this seems very complicated. I'm doing this ...
2
votes
1answer
81 views

How to extract ngrams from ambigous text after lemmatization?

After lemmatization of text I have a sequence of sets of lemmas, because every word can correspond to more than one lemma. How should I extract ngram statistics based on that? The only thing that ...
1
vote
1answer
80 views

Independence assumption in maximum entropy models in NLP

I am reading Klein and Manning's notes on Maximum Entropy for Natural Language Processing. On slide 22, they have an equation saying, $P(C|D,\lambda) = \Pi _{(c,d)\in (C,D)} P(c|d,\lambda)$. I am not ...
3
votes
0answers
96 views

Comparing term-frequency distributions with unequal sample sizes?

Background I have several datasets of word frequencies where some datasets have much more data than others: from 3000 samples to 20000 samples. I also have large reference corpora with millions of ...
0
votes
1answer
154 views

Regarding the feature generation method with SVM-based classification method

When using SVM to build classifier for a collection of documents, we can use term occurrence, term frequency or even TF/IDF. I would like to know what are the main disadvantages of using term ...
2
votes
0answers
20 views

what is the way of dealing with textual valued feature vectors for classification task?

I aim to work on twitter data for sentiment analysis but I am curios for the way of dealing such a huge number of textual features (words). Is using the Bag-Of-Words approach is the best? However I've ...
1
vote
0answers
80 views

Using sentiment lexicons or all words processing for sentiment analysis?

I am learning sentiment analysis to apply it to twitter real time data to predict user's mood. I ponder about using which alternative way to do that data mining job. Use all words to process and ...
3
votes
1answer
240 views

Latent Dirichlet allocation Implementation

I'm looking for some LDA implementation. I know about this one, MALLET but it is coded in Java and I need some more performant. Can someone give me some reference?
5
votes
1answer
576 views

Why is tf-idf used in conjunction with SVMs for classifying documents?

Term frequency - inverse document frequency is term count within a document weighted against the term's ubiquity within the corpus. This weight is based on the principle that terms occurring in ...
2
votes
0answers
78 views

Using n-grams to find data that does not 'belong'

Recently I posted a question over in CS.SE dealing with methods of classifying data. Essentially the problem is that I have a collection of strings (100's of thousands). Most of these strings are ...
2
votes
1answer
61 views

How to update a dynamic language model dataset?

I'm a statistics novice and I need help with a natural language problem. I'm writing a word-prediction algorithm for a mobile app. I'm using a unigram language model of word/count pairs where count ...
6
votes
2answers
1k views

Topic models and word co-occurrence methods

Popular topic models like LDA usually cluster words that tend to co-occur together into the same topic (cluster). What is the main difference between such topic models, and other simple ...
4
votes
0answers
71 views

Language modeling: why is adding up to 1 so important?

(if this venue is inappropriate, feel free to migrate it) In many natural language processing applications such as spelling correction, machine translation and speech recognition, we use language ...
0
votes
1answer
94 views

Lucene-based text feature construction

When doing the feature construction for text mining, does Lucene has a better performance in terms of classification/clustering result than the traditional bag-of-word approach?
2
votes
1answer
407 views

Different size of vocabulary made by Weka and R's tm

I own around 40,000 text files for preprocessing (in purpose of document classification). I used R (with tm package) for prototype and now looking for a equivalent Java library for products. ...
1
vote
4answers
873 views

Software or libraries to create doc-term matrix

does anyone know some Java libraries to create the document-term matrix for a large number (50,000) of documents ? I wish this library encompasses preprocessing functionalities, like stop-word and ...
2
votes
0answers
105 views

Feature construction for text mining

In the text mining, besides N-gram model, what are the state-of-art models for building feature space while capturing the dependence among the different words, or capturing the semantic meaning in the ...
1
vote
1answer
75 views

Larger ngrams vs nested ngrams?

Whenever I see people using ngrams - I often see them looking for higher chains like 4-grams, 5-grams and so on. However, I'm wondering why I never see any mention of "nested ngrams" (I'm not sure ...
4
votes
1answer
530 views

Regarding using bigram (N-gram) model to build feature vector for text document

A traditional approach of feature construction for text mining is bag-of-words approach, and can be enhanced using tf-idf for setting up the feature vector characterizing a given text document. At ...
2
votes
1answer
212 views

Software packages that can construct feature representation for a given text file using N-gram model

Are there any open source software packages, (Java, Matlab, R) that can generate a feature representation for a text document using N-gram model?
1
vote
1answer
278 views

Regarding the R packages that share the similar functionalities of NLTK toolkit

Are there any R packages that share the similar functionalities of NLTK toolkit?
1
vote
0answers
30 views

The approach of labelling a collection of documents using a shared topic

I have a collection of documents, and know they may share a single topic. Is there a way to identify this topic? I know LDA (Latent Dirichlet Allocation) is an approach. But LDA result is to associate ...
2
votes
0answers
66 views

Statistical model of a website

I know that HMMs can be used to construct statistical models of text. Thus, we can generate text according to this model, and compute the likelihood of a text sample under the model. What tools are ...
5
votes
1answer
250 views

At what n do n-grams become counterproductive?

When doing natural language processing, one can take a corpus and evaluate the probability of the next word occurring in a sequence of n. n is usually chosen as 2 or 3 (bigrams and trigrams). Is ...
14
votes
2answers
604 views

Why does Natural Language Processing not fall under Machine Learning domain? [closed]

I encounter it in many books as well as web. Natural Language Processing and Machine Learning are said to be different subsets of Artificial Intelligence. Why is it? We can achieve results of Natural ...
5
votes
1answer
755 views

TF-IDF cutoff percentage for tweets

I'm currently trying to analyze Tweets and classify them as either positive, negative, or neutral using the NLTK library in Python. I can see that there's potential in the approach that I'm taking, ...