Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

0
votes
0answers
12 views

Is it possible to use WEKA in a web based application? [on hold]

I am building a web based text mining application. For a word that user enters, the application has to: search it in google gather the documents pre-process by using Bag of words model cluster ...
-1
votes
0answers
12 views

Clustering headlines without full text [on hold]

I have a data-set with headlines of news. The number of topics is 100. Also I have a tf-idf file, which is not generated by me. I saw data-set, and I can determine something like 50 clusters. 50 - ...
0
votes
0answers
19 views

A document similarity measuring question

My question is about the below (quite straightforward) approach to measure similarity between two documents or texts or strings. How do they call this approach, this class of similarity measure? ...
0
votes
0answers
26 views

Latent Dirichlet Allocation yields different posterior distribution than simple Bayesian model

Method A: out of the box LDA I am using a package to run LDA on a sample of size m with n words in the vocabulary. The end ...
1
vote
0answers
20 views

Use tm_filter to search for multiple words

I´m new to R, so please bear with me. So, I know I can use the following to search for a word in several documents. ...
0
votes
1answer
12 views

How to continuously computate of tf-idf for relevance of single terms

I have a document corpus containing over 4 million documents. Now I want to build an index over terms from the documents of the corpus. Based on the tf-idf of these terms, I want to remove the least ...
1
vote
1answer
27 views

Naive Bayes Classifier in R with class weights

I'm searching for a Naive Bayes classifier in R where I can add a paramter for class weights. I need this, because my data is highly unbalanced. Eg.: Class1: 1000 examples Class2: 800 examples ...
0
votes
0answers
15 views

Meaning of SVD plot of $U$ and $V^T$

I am using SVD/PCA for text mining purposes. Having a $(|terms|,|documents|)$ normalized matrix $M$, by applying SVD, I should be able to reduce the dimensionality and just keep the most meaningful ...
1
vote
1answer
21 views

Is multiple stage binary classification a good idea if you have very few positives?

The problem is the following: We have a set of, say 5000 documents, with a single binary label. Say that 4900 documents are negative and only 100 are positive. I built a binary classifier while ...
1
vote
0answers
18 views

why add one in inverse document frequency

My textbook lists the idf as $log(1+\frac{N}{n_t})$ where $N$: Number of Documents $n_t$: Number of Documents containing term $t$ Wikipedia lists this formula as a smoothed version of the actual ...
1
vote
1answer
31 views

How to use TF-IDF for features selection in Text classification?

I have a small confusion regarding TFIDF. I am planning to use TFIDF for creating better word dictionary to be used in Naive Bayes classifier. I am calculating the TDIDF of all words in respective ...
0
votes
0answers
23 views

Typical range of values for TFIDF

I am working on a text corpus. Each line contains between 10 and 50 words. There are around 25 000 words in the whole text and 1 000 000 lines. I turned this corpus into its tf-idf representation. I ...
0
votes
0answers
15 views

What is VectorSource and VCorpus in 'tm' (Text Mining) package in R

I'm not quite sure what exactly VectorSource and VCorpus are in 'tm' package. The documentation is unclear on these, can anyone make me understand in simple terms?
1
vote
1answer
21 views

Help: Text Mining + Classification - From customer comments to predicted solution [closed]

I have a data set that consists of the information generated by a service call for a home appliance. The data set consists of a column with the sentence of the customer's complaint and a ...
0
votes
0answers
10 views

Deep classification, training vs inference phase

As I have described in Deep classification, how to represent category as TF-IDF vector?. I am trying to understand more in detail and reproduce the Deep classification for large scale taxonomy ...
1
vote
0answers
17 views

Deep classification, how to represent category as TF-IDF vector?

I am trying to implement so called deep classification method described here. I am trying to replicate chapter 4.2, with category-based search. Unfortunately, I am not sure how should I represent the ...
2
votes
0answers
99 views

Finding idf only for text mining

We find tf-idf for training phase in text mining, however, in test phase, we need the tf for each element in test set, but should use idf in train set, so is there any api in python that can calculate ...
0
votes
0answers
22 views

tf-idf in text mining

I used sklearn of Python for getting tf-idf attribute in text analysis, but the problem is: I have about 78000 words in train_set, but the tf-idf matrix only has 39000 words. What is the problem ...
0
votes
0answers
20 views

IKAnalyzer in text mining

Does anyone use IKAnalyzer for word segmentation in the preprocess for text mining? I have never loaded my own extended dictionary or stopword dictionary successfully. The following is the ...
2
votes
1answer
74 views

How to set the dictionary for text analysis using neural networks

I want to use a neural network to do text analysis. If I use a large dictionary, then it will contain all the words in training and test set, but the size of the dictionary is too large which will ...
0
votes
1answer
40 views

Distinguishing objective from subjective text using a Naive Bayes classifier

I am trying to built a classifier for subjective and objective text using imdb data. For objective data point I am using the movie's plot summary as input. For subjective data points I am using ...
1
vote
2answers
23 views

Text Categorization packages in R [closed]

I have a dataset of 1400 data points. My fields are Description and Category. I have 1200 data points as the training dataset and 200 for testing purpose. My goal is to analyze the Description column ...
0
votes
0answers
21 views

Need help in understanding variable

I recently started working on a data set where the main objective is to predict number of shares an article gets. The data set has many variables such as number of words,number of images etc. I am ...
3
votes
1answer
69 views

Understanding the use of logarithms in the TF-IDF logarithm

I was reading: https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Definition But I cannot seem to understand exactly why the formula was constructed the way it is. What I do Understand: iDF should at ...
0
votes
1answer
22 views

What's meaning of BOS and EOS in CRFSuite feature list and what is the role of them?

In NER(Named Entity Recognition) example in python-crf package website we see this function as feature generator: ...
0
votes
2answers
321 views

How to use the `NGramTokenizer` from `tm` to build a term document matrix?

I installed the tm library and want to build n-grams of a corpus using the NGramTokenizer from the ...
2
votes
1answer
61 views

Why is random forest inconsistent in text mining?

Earlier I've used SVM (rbf kernel) in text mining with success, and after that for similar text mining work with long texts I've used random forest with success as well. However in a recent kaggle ...
1
vote
1answer
45 views

Use of a bagging model or feature engineering?

As a pet project, I have been learning some data analysis and machine learning skills (mainly text analytics) with the Analytics Edge course on edX. I decided to put some of my new skills at use ...
1
vote
1answer
55 views

Is this interpretation of sparsity accurate?

According the documentation of the removeSparseTerms function from the tm package, this is what sparsity entails: ...
0
votes
0answers
22 views

How do I use the Stanford dependency parser and constituency parser to extract attribute value pairs from product descriptions?

I am doing a small project currently where I have to extract attribute-value pairs from product descriptions taken from the web. So I have been trying out various methods and my latest idea is: Use ...
0
votes
0answers
20 views

Classify inventory part names into cost categories

I'm wondering if it's possible to do, and if so, how would I do it? I would like to create a model that could classify part names (inventory part names) to cost categories ('under \$1', '\$1 to 9.99', ...
0
votes
1answer
73 views

topic similarity semantic PMI between two words wikipedia

I am trying to compute pointwise mutual information (PMI) using wikipedia as data source. Given two words, PMI defines the relation between two words. The formula is as below. ...
0
votes
1answer
32 views

How do I improve the accuracy of my supervised document classification model? [closed]

Given 1000 legal judgement documents, 900 of which are labeled, my task is to predict the label for the remaining 100 documents. The labeled documents belong to 41 different categories of Law, with ...
1
vote
1answer
49 views

Incorporating new words in tfidf feature-vector for online clustering

I am building an Online news clustering system using Lucene and Mahout libraries in java. I intend to use vector space model and tfidf weights for Kmeans(or fuzzy/streamKmeans). My plan is : Cluster ...
1
vote
1answer
65 views

Measure the similarty between two sequences of letters

I'm trying to measure the similarity between two time-series sequences of letters with different lengths (e.g. s1=[A;A;A;C;B], s1=[Q;A;A;A;A;A] ). The order is very important. (e.g. s3=[A;A;A;C;C;C;C] ...
0
votes
0answers
34 views

Finding probability distribution of topics from LDA for unseen documents

How can I train LDA (topicmodels library in R) upon a corpus of documents to find the topics and then for a previously unseen set of documents get probabilities with which each document has the above ...
1
vote
1answer
203 views

LDA with tm package in R using bigrams

I have a csv with every row as a document. I need to perform LDA upon this. I have the following code : ...
1
vote
1answer
45 views

Does one need to adjust for document length (in terms of pages) in topic modeling?

I am thinking about whether one needs to normalize or weight a topic model by document length (page length)? I am estimating a topic model using social science (JSTOR) articles, where they vary in ...
4
votes
3answers
418 views

Text Mining: how to cluster texts (e.g. news articles) with artificial intelligence?

I have built some neural networks (MLP (fully-connected), Elman (recurrent)) for different tasks, like playing Pong, classifying handwritten digits and stuff...additionally I tried to build some first ...
2
votes
0answers
81 views

Text mining: Robust correlation or similarity measures

I'm currently using word_cor function (qdap package). I observed that the function is not robust as it implements Pearson, Spearman and Kendall measures only: non-occurrence of both words (in the ...
4
votes
1answer
34 views

How to prove that text is linearly separable?

I sentiment analisys task, for this I used SVM with an rbf kernel and a linear one. The results for the linear kernel were better than the rbf, from this I know that text is linearly separable, but ...
0
votes
0answers
24 views

What kind of feature selection do I need for text mining?

I have a data set of questions belonging to 10 different categories namely (definitions, factoids, abbreviations, fill in the blanks, verbs, numerals, dates, puzzle, etymology and category relation). ...
2
votes
1answer
52 views

Estimating the best length of n-gram

I have a long sequence of words or letters {word1 word2 word3 word1 word1 word2 ..etc}. Lets say we extract all the ngrams (unigrams, bigrams, trigrams, 4-gram, 5-gram ....) along with their frequency ...
2
votes
2answers
66 views

How can I improve feature selection for my Naive Bayes Classifier?

I am classifying companies into two classes ( a particular business type, or not that business type ), using a Naive Bayes Classifier. Specifically, I'm using PHP and PHP NLP Tools. I have two ...
1
vote
0answers
28 views

Best practices to compute TFIDF matrix based on another TFIDF matrix in R

I'd like to compute a TFIDF matrix (tfidf_matrix_b) based on a previously computed TFIDF matrix (tfidf_matrix_a). Is there a ...
1
vote
0answers
21 views

Text mining of machine logs to find correlation between errors in R [duplicate]

I've with me 50 MB data from a machine consisting of event logs such as device status, warning and error. I wish to perform text mining on the same to find correlation between errors i.e. one error ...
-2
votes
1answer
30 views

How to do text clustering for a set of around 10000 messages?

I have around 10000 messages in a variable, i want to form clusters of them based on similarity, so that I can assign some class say 1-10, if 10 clusters are formed and run analysis on them. How can ...
9
votes
2answers
525 views

Bag-of-Words for Text Classification: Why not just use word frequencies instead of TFIDF?

A common approach to text classification is to train a classifier off of a 'bag-of-words'. The user takes the text to be classified and counts the frequencies of the words in each object, followed by ...
1
vote
1answer
127 views

Text analysis : What after term-document matrix?

I am trying to build predictive models from text data. I built document-term matrix from the text data (unigram and bigram) and built different types of models on that (like svm, random forest, ...
0
votes
0answers
29 views

machine learning for a ontology classification problem

I am working on a ontology based classification problem.The main objective was: computing ontology has keywords related to different categories.Each category talks about the domain it is related.For ...