Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

1
vote
1answer
14 views

Text Classification using TfIdf and Bernoulli NB

So, as I am reading about Bernoulli distribution and text classification, I want to understand how Bernoulli uses TfIdf features? Since TfIdf values are within [0-1) but Multivariate Bernoulli assumes ...
0
votes
0answers
16 views

Identifying topic name

When we perform Latent Dirichlet allocation on a set of documents to cluster, just assume give n=5 topics. How can we assign the topic name for these topics after performing LDA? I would like to give ...
0
votes
0answers
18 views

How to find the perplexity of a corpus

The formula of the perplexity measure is thus: $ p: \left(\frac{1}{\sqrt[n]{p(w_1^n)}}\right) $ where: $p(w_1^n)$ is: $\prod_{i=1}^n p(w_i)$ If I understand it correctly, this means that I ...
0
votes
1answer
24 views

A single document as input to LDA?

We use topic modelling usually on a collection of documents - which makes the input. But what if I only have a single document where I want to see the underlying topics in it? I have heard that you ...
0
votes
1answer
22 views

Multiple labels in supervised learning algorithm

I have a corpus of text with a corresponding topics. For example "A rapper Tupac was shot in LA" and it was labelled as ...
2
votes
1answer
23 views

determining significance of term use

Thing one: feel free to RTFM me: I'm definitely looking for search-able terms or background reading. Our situation is this: we have a set of 140 reviewers and 20 elements. Each reviewer reviews each ...
0
votes
0answers
17 views

The best algorithm for short documents clustering

I have a corpus of short text documents. Each document is an automatic recognized phone conversation (a dialog) from a large call center. The texts are not clean and have lots of grammar and other ...
5
votes
3answers
261 views

How would you categorize / extract information out of job descriptions?

I have a bunch of job descriptions entered by users. There are all sort of misspells and bad data. i.e: ...
1
vote
1answer
45 views

How to prepare a dataset for text classification

I would like to compare some algorithms for performing sentiment classification (Naive Bayes, SVM, and ...
0
votes
0answers
35 views

VW multiclass classification

I am new to vw and trying to do a multiclass text classification with 18 classes. features are unigram, bigram and trigrams. Total features are around 1.4 million Total training examples 35 million ...
2
votes
0answers
15 views

How to measure how 'well' I am matching Google keywords?

For google keywords you can bid on a broad match. For example let's say I bid on the keyword 'best hamburger' and somebody searches 'What sort of beef makes the best hamburger?' and 'eat best ...
0
votes
1answer
53 views

In natural language processing (NLP), how do you make an efficient dimension reduction?

In NLP, it's always the case that the dimension of the features are very huge. For example, for one project at hand, the dimension of features is almost 20 thousands (p = 20,000), and each feature is ...
1
vote
0answers
13 views

Calculate association between text documents [duplicate]

I've got 6000 reports. For each report, I've got a vector with the keywords in it and a vector with the tokens of its abstract. Now I want to calculate some association between those two sets. Is ...
0
votes
1answer
38 views

Vector Space Model for Online News Clustering

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common ...
2
votes
0answers
48 views

Has the reported state-of-the-art performance of using paragraph vectors for sentiment analysis been replicated?

I was impressed by the results in the ICML 2014 paper "Distributed Representations of Sentences and Documents" by Le and Mikolov. The technique they describe, called "paragraph vectors", learns ...
0
votes
1answer
17 views

Easy way of indicating the likelihood of a text?

I am trying to find out a simple way of quantifying "how much sense a text paragraph makes". The measure is not necessarily fail-proof, but it has to be somewhat simple. One example measure that ...
3
votes
1answer
72 views

Methodology for standardizing names

We have a great amount of data with user generated names for company names (think of bills where the company wrote their own name). In our data cleaning phase, we clustered the different companies ...
0
votes
0answers
24 views

Dataset with mixed structured/unstructured data

I am looking for some mixed-type dataset with free-form textual features along with some structural features (preferably continuous) - labeled, for supervised setting (classification or prediction). ...
1
vote
2answers
85 views

How to transform test set to the PCA space of the training set, if the features in train and test are different?

I'm working on a text classification project, and I want to reduce the tf-idf matrix dimension with Principal Component Analysis (PCA) and then train my model with this, which is pretty ...
3
votes
0answers
27 views

TF-IDF Matrix and Regression

I am trying to build a regression model based on some tweets that my company put on our company feed. I would like to transform all of the tweets, and use them to tell me which word(s) were most ...
1
vote
0answers
14 views

Is Social Network Analysis or NER the best way to create a semantic graph?

I am planning to create a semantic graph by creating an automatic ontology. I want to know which is the best process to do it. Doing social network analysis to create people, relationships, likes, ...
1
vote
0answers
99 views

Any advice on how to improve my accuracy rate in text classification?

I'm trying to do a text classification task. Here are some specs: Context file size = 1M+ documents already labeled Number of top-labels = 17 Number of sub-labels = around 130 Each document is ...
0
votes
0answers
22 views

Latent Semantic Analysis: scale representation of documents?

After performing SVD on the term-document matrix, the right eigenvectors correspond to the representation of documents in the reduced concept space. In order to use this for say text classification, ...
0
votes
1answer
197 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
0
votes
0answers
32 views

R Clustering Evaluation (Adaptive Kmeans)

i know there are several threads about this topic, but most i read, most i get confused. I'm doing a project that consists in clustering some data (news articles). I used adaptive Kmeans ...
3
votes
1answer
117 views

can we generate a random words from English letters that follow the bigram of the English language

The main issue is that several research building their solution of detecting and classifying English language is based on bigram distribution. However, I would like to know if it possible to generate ...
1
vote
0answers
69 views

In Kneser-Ney smoothing, how are unseen words handled?

From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as $ \begin{align} P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
1
vote
0answers
95 views

Naive Bayes and text classification: which probability model and vectorizer combination makes sense?

I am wondering which combinations of Naive models can be paired with different vectorizing methods so that it makes sense. Let's say we have a simple binary spam-classification task. Multinomial ...
0
votes
0answers
24 views

choosing best value for N when using N-Gram approach

the question is quite general, but I am doing a research related to supervised machine learning to classify two set of characters into two categories. in fact, I want to compute some measures of ...
0
votes
1answer
114 views

Using topic words generated by LDA to represent a document

I want to do document classification by representing each document as a set of features. I know that there are many ways: BOW, TFIDF, ... I want to use Latent Dirichlet Allocation (LDA) to extract ...
0
votes
0answers
47 views

Unsupervised machine learning with numerical and text data

My dataset has numerical variables and one "Note" variable with around a paragraph of text. I'd like to classify the data with an unsupervised learning algorithm. I've done some searching and one ...
0
votes
0answers
20 views

Asking tweet classification

I want to ask you the process to classify the tweet data. Now, I am working to Twitter data but i have confuse how to classify the tweet data using Mallet Tool. Example; I have 200,000 tweets. The ...
1
vote
0answers
21 views

Extracting city name from free text?

I'm having a set of free text from web. Since the users type their location in that field, we have many un-normalized city names. For example, "Shanghai, China" "China, ShangHai" might mean the ...
0
votes
0answers
14 views

How do I extract a keyword plus its corresponding number out of a text

I am finding my way around in the different text mining tools, but I can't find the technique that does the following: Extract out of the follwing advertisement texts ...
0
votes
1answer
76 views

Using a Dictionary for Text Mining

I want to use a dictionary to analyze a text with text mining mechanisms. So I'm looking for a dictionary with scores for each words, so for example the word "cool", +1, gets a positive score, the ...
0
votes
0answers
13 views

Mallet API installation

I am trying to use the mallet API to classify text. Previously created classifier using the command line tool. I don't know to use Mallet packages using Java. Thank you
0
votes
0answers
5 views

Get positions/values from heat costs bill

for a project I need to extract values from customers yearly heat costs bill. The customer takes a photo of the bill and the program should extract the values heating period of the billing, type of ...
1
vote
0answers
30 views

CoreNLP SemanticGraph - search for edges with specific lemmas

I'm using Stanford CoreNLP's dependency parser, and wondering how to make a generic search for SemanticEdge(s) with specific head lemma, dependent lemma, and lexical relationship. For example, if I ...
0
votes
0answers
8 views

automatic assign class name based on text

My question is , I have a set of plain text , i want to create category based on the text. Eg: i have written something about Soup recepie then the algorithm must create a category called Food. After ...
0
votes
1answer
106 views

Multi-class Confusion Matrix to Binary confusion matrix

i know the main concepts of data/text mining but i used them mainly in binary classification problems (just two classes). i am now dealing with a problem with 8 classes and i am atruggling how to ...
0
votes
0answers
74 views

Using relative frequency for Euclidean and cosine distance (dissimilarity)

How to calculate the Euclidean distance (dissimilarity) between two documents, e.g., D1 and D2 using relative frequency? Here is an example of both cosine and Euclidean distance between two ...
2
votes
0answers
27 views

Finding related words

I have several files, each of which contains unique terms which are related to each other(without sentence structure). So for finding the word relationships I created a dictionary of bi-grams for ...
2
votes
2answers
78 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
0
votes
0answers
26 views

Categorizing text into IPTC subjects

Does anyone know where I can find a corpus I can use to train a classifier into IPTC news categories (http://www.iptc.org/site/NewsCodes/) ? A google search was not very useful. Thank you
0
votes
1answer
38 views

predict category by using K-NN algorithm having text features

I would like to predict the category of the provided data by using K-NN algorithm. Here is an example of the training data set ...
2
votes
2answers
52 views

Dimensionality reduction (PCA) for plotting text documents on a graph

I have 50 text documents There are 500 possible words, after a stop list has been applied My term/document sparse matrix is therefore 50x500 I'd like to cluster these documents. One easy way to do ...
1
vote
1answer
71 views

SVM Classification with Duplicate Training Instances

I'm using SVMs with linear kernel for sentence classification (binary). My dataset contains many duplicate instances i.e. many sentences in the training set have identical feature vectors. In the ...
0
votes
0answers
27 views

Post-process the output of a Multinomial Naive Bayes text classifier

I have a multinomial text classification application where there are other features than the words in text which can be useful to do the classification e.g, contains email address, contains an URL, ...
3
votes
3answers
228 views

In Naive Bayes, why bother with Laplacian smoothing when we have unknown words in the test set?

I was reading over Naive Bayes Classification today. I read, under the heading of Parameter Estimation with add 1 smoothing: "Let $c$ refer to a class (such as Positive or Negative), and let $w$ ...
1
vote
0answers
31 views

Power Analysis for Text Mining

I have a population of 6 million text files with which I want to perform sentiment analysis (and text analysis more generally). I will need to manually hand code a subset of these texts into positive ...