Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

0
votes
0answers
6 views

Mallet API installation

I am trying to use the mallet API to classify text. Previously created classifier using the command line tool. I don't know to use Mallet packages using Java. Thank you
0
votes
0answers
10 views

Asking Mallet Language Tool Kit [migrated]

I am working to Text Document. I try to filter the document using Mallet Language Tool Kit. My questions are. 1. I use macintosh os. should i install ant ? 2. How can i use the packages in this ...
0
votes
0answers
4 views

Get positions/values from heat costs bill

for a project I need to extract values from customers yearly heat costs bill. The customer takes a photo of the bill and the program should extract the values heating period of the billing, type of ...
1
vote
0answers
14 views

CoreNLP SemanticGraph - search for edges with specific lemmas

I'm using Stanford CoreNLP's dependency parser, and wondering how to make a generic search for SemanticEdge(s) with specific head lemma, dependent lemma, and lexical relationship. For example, if I ...
0
votes
0answers
8 views

automatic assign class name based on text

My question is , I have a set of plain text , i want to create category based on the text. Eg: i have written something about Soup recepie then the algorithm must create a category called Food. After ...
0
votes
1answer
26 views

Multi-class Confusion Matrix to Binary confusion matrix

i know the main concepts of data/text mining but i used them mainly in binary classification problems (just two classes). i am now dealing with a problem with 8 classes and i am atruggling how to ...
0
votes
0answers
34 views

Using relative frequency for Euclidean and cosine distance (dissimilarity)

How to calculate the Euclidean distance (dissimilarity) between two documents, e.g., D1 and D2 using relative frequency? Here is an example of both cosine and Euclidean distance between two ...
2
votes
0answers
21 views

Finding related words

I have several files, each of which contains unique terms which are related to each other(without sentence structure). So for finding the word relationships I created a dictionary of bi-grams for ...
2
votes
2answers
75 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
0
votes
0answers
6 views

Categorizing text into IPTC subjects

Does anyone know where I can find a corpus I can use to train a classifier into IPTC news categories (http://www.iptc.org/site/NewsCodes/) ? A google search was not very useful. Thank you
0
votes
1answer
24 views

predict category by using K-NN algorithm having text features

I would like to predict the category of the provided data by using K-NN algorithm. Here is an example of the training data set ...
2
votes
2answers
31 views

Dimensionality reduction (PCA) for plotting text documents on a graph

I have 50 text documents There are 500 possible words, after a stop list has been applied My term/document sparse matrix is therefore 50x500 I'd like to cluster these documents. One easy way to do ...
1
vote
1answer
36 views

SVM Classification with Duplicate Training Instances

I'm using SVMs with linear kernel for sentence classification (binary). My dataset contains many duplicate instances i.e. many sentences in the training set have identical feature vectors. In the ...
0
votes
0answers
19 views

Post-process the output of a Multinomial Naive Bayes text classifier

I have a multinomial text classification application where there are other features than the words in text which can be useful to do the classification e.g, contains email address, contains an URL, ...
3
votes
3answers
59 views

In Naive Bayes, why bother with Laplacian smoothing when we have unknown words in the test set?

I was reading over Naive Bayes Classification today. I read, under the heading of Parameter Estimation with add 1 smoothing: "Let $c$ refer to a class (such as Positive or Negative), and let $w$ ...
1
vote
0answers
19 views

Power Analysis for Text Mining

I have a population of 6 million text files with which I want to perform sentiment analysis (and text analysis more generally). I will need to manually hand code a subset of these texts into positive ...
1
vote
0answers
25 views

Binary classification of dated text documents with seasonality

I have a collection of training documents with publication dates, where each document is labeled as belonging (or not) to some topic T. I want to train a model that will predict for a new document ...
0
votes
0answers
48 views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
1
vote
1answer
84 views

Feature selection : how to select the Information Gain threshold?

I am trying to use Information Gain to select features when classifying text with a Support Vector Machine. For each word in our training data, we computed its information gain. Then, we should keep ...
0
votes
1answer
20 views

How to increase a particular terms's weightage?

I am doing Text classification using LibSVM in Rapid Miner. I am using TFIDF values for processing documents. I need to Increase weightage of some terms in the documents(for eg. words in BOLD and ...
0
votes
0answers
25 views

How to collect related words with specified one?

Assume I have a word such as statistics, and research articles or text books about statistics. My question is how to detect top 100 related, highly associated ...
0
votes
0answers
34 views

How to implement data I have to svmtrain() function in MATLAB?

I have to write a script using MATLAB which will classify my data. My data consists of 1051 web pages (rows) and 11000+ words (columns). The first 230 rows are about computer science course (to be ...
0
votes
0answers
34 views

Hierarchical text classification in R

I am working Automatic Email routing. My historical data set has Email Description,Division and Category. There are around 20 division and 150 categories. I have built one vs. all classifier ...
2
votes
3answers
152 views

Naive Bayes: Imbalanced Dataset in Real-time Scenario

I am using scikit-learn Multinomial Naive Bayes classifier for binary text classification (classifier tells me whether the document belongs to the category X or not). I use a balanced dataset to train ...
1
vote
1answer
33 views

Assigning meaningful cluster name automatically

The objective of my work is to cluster the text documents. Once the documents are clustered, traditionally the system will assign numeric value for the clustered group. For example if I have 5 ...
1
vote
1answer
36 views

How do I perform an IDF calculation?

How do I perform an IDF calculation? I am uncertain as to whether IDF should be calculated in per-class level or for the entire document set (that contains multiple classes).
0
votes
1answer
39 views

I have an easy Text mining frequency question

I have a very simple question. I am doing a simple frequency analysis where I am looking at two documents and comparing the frequency of the top used words in each document. I want to do a comparison ...
2
votes
1answer
46 views

Selecting number of clustering classes automatically

I am working in text clustering. I would like to find a way to identify the number of classes for the clustering process automatically rather than proving the number of class manually. Is their any ...
0
votes
0answers
31 views

What are the mathematics I need to learn, before I start research in data mining [duplicate]

I usually use text mining, graph mining, Information retrieval, and natural lanuage processing. Also i will use the fundamental concepts of data mining like classification, association and clustering. ...
0
votes
1answer
72 views

Comparing topic distributions between corpora using Latent Dirichlet Allocation and R topicmodels or python gensim

So I am working on a problem where I want to extract a set of LDA topics from one corpus, and then compare the distribution of those topics in other corpora. So basically I want to lock-in the topics ...
1
vote
1answer
80 views

Create a matrix of tf-idf values from documents

I have a set of documents like: D1 = "The sky is blue." D2 = "The sun is bright." D3 = "The sun in the sky is bright." and a ...
0
votes
0answers
26 views

In which Data Stream Mining Algorithms do Damped Windows make sense?

For Data Stream Mining, especially in Document Classification, the most common ML algorithms are Multinomial Naive Bayes, Stochastic Gradient Descent and Ozbag (ADWIN). When looking at their ageing ...
0
votes
0answers
51 views

Training a multiclass SVM

I am using Support Vecotr Machine(SVM) with 4 Class. My corpus contain 185 documents with 4 different subjects. For each subject I defined a profile with 3 or 4 keywords (of course they are separated ...
0
votes
1answer
26 views

How to measure the similarity among categories?

Suppose we have several documents. These docs are classified into several categories. But there could be issues like these: The categories may not be properly defined. Or The categories are properly ...
3
votes
3answers
115 views

Alternatives to bag-of-words based classifiers for text classification?

Most of the text classifiers are based on the bag-of-words approach where you loose the context that a particular word appears. As a solution (or simple solution?) we can use n-grams as features. But ...
1
vote
0answers
35 views

Hierarchiqual prediction using R

I'm pretty new in R, and I couldn't find any information about a package who can do the following: supposing that I have a set of data (for instance, different text documents), which can have several ...
0
votes
0answers
16 views

What does Language Model look like?

I am working on a machine learning + NLP project. The corpora is from a very specific domain. Someone tells me I need a language model for that specific domain. So I decide to build one myself since ...
0
votes
1answer
60 views

Clustering structured data: Assessing the similarity of documents that appear in tree structure

Usually when performing text document clustering, similarities across documents are assessed based on the lexical content of documents. But, in my problem, I wish to consider both the lexical content ...
0
votes
0answers
167 views

Calculating TF-IDF in Matlab for Stop Words

I have been calculating tf-idf for removing stop words in my file. I am experiencing problems. First let us have a look at code I have written.. ...
1
vote
2answers
84 views

How can I analyze my incoming email?

I would like to analyze the email I receive in my Gmail inbox in order to systematically come up with effective Gmail filters for the most common types of email. I am prepared to manually curate and ...
0
votes
0answers
22 views

LexRank damping factor

I am looking into using LexRank to do some text summarization. I am looking at the original paper. One thing that puzzles me is whether a damping factor is used or not. The formulae are all using it, ...
2
votes
1answer
64 views

How can I evaluate the performance of a system that generates word clusters?

The word2vec tool uses deep learning to compute vector representations of words. They've mentioned that - "The word vectors can be also used for deriving word classes from huge data sets. This is ...
0
votes
2answers
71 views

What is the minimum training set size required for a given number of features for document classification?

For document classification problems, is there a rule of thumb for the number of training instances required for the number of terms in the vocabulary? I am using a logistic regression classifier ...
0
votes
0answers
38 views

Compare patterns of text in R

I would like to compare a number of patterns within a huge dataset. I have one variable that is in textform, i.e. 1 der tus wif 2 fiu seg suf erf utz tus wif ...
3
votes
1answer
80 views

Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
0
votes
1answer
89 views

How to perform text mining on an online news article?

I am new to data mining and currently working on an online news article from TOI. My aim is to get some useful information out of this text which is not clear when you read the article and the most ...
0
votes
1answer
28 views

Topic and subject classification

I have a set of documents that are OCR-ed and represented as a text file. I want to find out what are the documents that are talking about the same subject and maybe about the same person. I started ...
0
votes
1answer
38 views

Data Conversion to Standard data format in hierarchical Dirichlet process

I'm trying to test the performance of posterior inference on a set of documents with hierarchical Dirichlet process for topic modeling. How can i convert my data (document) to standard data format ...
3
votes
1answer
157 views

Machine learning techniques for spam detection, and in general for text classification

I am going to configure a system for spam detection. What I have is a dataset of labeled (spam/not-spam) strings containing, mostly, sentences. I have a background in machine learning techniques, but ...
0
votes
1answer
61 views

how to identify salient words after LSA?

I used Latent Semantic Analysis (LSA) to extract latent topics (i.e., the polynomials coefficient1*word1 + coefficient2*word2 + ...) from a certain corpus. I know that the larger the (absolute value ...