Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

0
votes
0answers
7 views

Deep classification, training vs inference phase

As I have described in Deep classification, how to represent category as TF-IDF vector?. I am trying to understand more in detail and reproduce the Deep classification for large scale taxonomy ...
0
votes
0answers
12 views

Deep classification, how to represent category as TF-IDF vector?

I am trying to implement so called deep classification method described here. I am trying to replicate chapter 4.2, with category-based search. Unfortunately, I am not sure how should I represent the ...
2
votes
0answers
31 views

Finding idf only for text mining

We find tf-idf for training phase in text mining, however, in test phase, we need the tf for each element in test set, but should use idf in train set, so is there any api in python that can calculate ...
0
votes
0answers
21 views

tf-idf in text mining

I used sklearn of Python for getting tf-idf attribute in text analysis, but the problem is: I have about 78000 words in train_set, but the tf-idf matrix only has 39000 words. What is the problem ...
0
votes
0answers
8 views

Splitting DocumentTermMatrix in R [migrated]

I'm looking to create a word pair prediction function, but am having trouble working with DocumentTermMatrix to data frame or similar to use in prediction function. Here is my working code: ...
0
votes
0answers
19 views

IKAnalyzer in text mining

Does anyone use IKAnalyzer for word segmentation in the preprocess for text mining? I have never loaded my own extended dictionary or stopword dictionary successfully. The following is the ...
2
votes
1answer
65 views

How to set the dictionary for text analysis using neural networks

I want to use a neural network to do text analysis. If I use a large dictionary, then it will contain all the words in training and test set, but the size of the dictionary is too large which will ...
0
votes
1answer
30 views

Distinguishing objective from subjective text using a Naive Bayes classifier

I am trying to built a classifier for subjective and objective text using imdb data. For objective data point I am using the movie's plot summary as input. For subjective data points I am using ...
0
votes
0answers
25 views

How to clean corpora from words with fourfold repeated letters? [migrated]

On inspecting a term-document matrix created with TermDocumentMatrix of the tm library (of Twitter data), I find that many words ...
1
vote
2answers
20 views

Text Categorization packages in R [closed]

I have a dataset of 1400 data points. My fields are Description and Category. I have 1200 data points as the training dataset and 200 for testing purpose. My goal is to analyze the Description column ...
-3
votes
0answers
17 views

What does keyword mean in text mining [duplicate]

I have encountered variables: **kw_max_min: Worst keyword (max. shares)**** in a data set. No description has been given about this variable and I am not able to comprehend what this variable mean.If ...
0
votes
0answers
18 views

Need help in understanding variable

I recently started working on a data set where the main objective is to predict number of shares an article gets. The data set has many variables such as number of words,number of images etc. I am ...
3
votes
1answer
52 views

Understanding the use of logarithms in the TF-IDF logarithm

I was reading: https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Definition But I cannot seem to understand exactly why the formula was constructed teh way it is. What I do Understand: iDF should at ...
0
votes
1answer
11 views

What's meaning of BOS and EOS in CRFSuite feature list and what is the role of them?

In NER(Named Entity Recognition) example in python-crf package website we see this function as feature generator: ...
0
votes
2answers
163 views

How to use the `NGramTokenizer` from `tm` to build a term document matrix?

I installed the tm library and want to build n-grams of a corpus using the NGramTokenizer from the ...
2
votes
1answer
48 views

Why is random forest inconsistent in text mining?

Earlier I've used SVM (rbf kernel) in text mining with success, and after that for similar text mining work with long texts I've used random forest with success as well. However in a recent kaggle ...
1
vote
1answer
38 views

Use of a bagging model or feature engineering?

As a pet project, I have been learning some data analysis and machine learning skills (mainly text analytics) with the Analytics Edge course on edX. I decided to put some of my new skills at use ...
1
vote
1answer
42 views

Is this interpretation of sparsity accurate?

According the documentation of the removeSparseTerms function from the tm package, this is what sparsity entails: ...
0
votes
0answers
19 views

“CFINDER” output data importation and calculation of mean of pairs [migrated]

I face a problem: I have to import a text file (from_soft.txt, which is output from "CFINDER" software). It looks like this: ...
0
votes
0answers
18 views

How do I use the Stanford dependency parser and constituency parser to extract attribute value pairs from product descriptions?

I am doing a small project currently where I have to extract attribute-value pairs from product descriptions taken from the web. So I have been trying out various methods and my latest idea is: Use ...
0
votes
0answers
19 views

Classify inventory part names into cost categories

I'm wondering if it's possible to do, and if so, how would I do it? I would like to create a model that could classify part names (inventory part names) to cost categories ('under \$1', '\$1 to 9.99', ...
0
votes
1answer
54 views

topic similarity semantic PMI between two words wikipedia

I am trying to compute pointwise mutual information (PMI) using wikipedia as data source. Given two words, PMI defines the relation between two words. The formula is as below. ...
0
votes
1answer
28 views

How do I improve the accuracy of my supervised document classification model? [closed]

Given 1000 legal judgement documents, 900 of which are labeled, my task is to predict the label for the remaining 100 documents. The labeled documents belong to 41 different categories of Law, with ...
1
vote
1answer
31 views

Incorporating new words in tfidf feature-vector for online clustering

I am building an Online news clustering system using Lucene and Mahout libraries in java. I intend to use vector space model and tfidf weights for Kmeans(or fuzzy/streamKmeans). My plan is : Cluster ...
1
vote
1answer
58 views

Measure the similarty between two sequences of letters

I'm trying to measure the similarity between two time-series sequences of letters with different lengths (e.g. s1=[A;A;A;C;B], s1=[Q;A;A;A;A;A] ). The order is very important. (e.g. s3=[A;A;A;C;C;C;C] ...
0
votes
0answers
22 views

Finding probability distribution of topics from LDA for unseen documents

How can I train LDA (topicmodels library in R) upon a corpus of documents to find the topics and then for a previously unseen set of documents get probabilities with which each document has the above ...
1
vote
1answer
109 views

LDA with tm package in R using bigrams

I have a csv with every row as a document. I need to perform LDA upon this. I have the following code : ...
1
vote
1answer
39 views

Does one need to adjust for document length (in terms of pages) in topic modeling?

I am thinking about whether one needs to normalize or weight a topic model by document length (page length)? I am estimating a topic model using social science (JSTOR) articles, where they vary in ...
4
votes
3answers
308 views

Text Mining: how to cluster texts (e.g. news articles) with artificial intelligence?

I have built some neural networks (MLP (fully-connected), Elman (recurrent)) for different tasks, like playing Pong, classifying handwritten digits and stuff...additionally I tried to build some first ...
2
votes
0answers
49 views

Text mining: Robust correlation or similarity measures

I'm currently using word_cor function (qdap package). I observed that the function is not robust as it implements Pearson, Spearman and Kendall measures only: non-occurrence of both words (in the ...
4
votes
1answer
27 views

How to prove that text is linearly separable?

I sentiment analisys task, for this I used SVM with an rbf kernel and a linear one. The results for the linear kernel were better than the rbf, from this I know that text is linearly separable, but ...
0
votes
0answers
23 views

What kind of feature selection do I need for text mining?

I have a data set of questions belonging to 10 different categories namely (definitions, factoids, abbreviations, fill in the blanks, verbs, numerals, dates, puzzle, etymology and category relation). ...
2
votes
1answer
48 views

Estimating the best length of n-gram

I have a long sequence of words or letters {word1 word2 word3 word1 word1 word2 ..etc}. Lets say we extract all the ngrams (unigrams, bigrams, trigrams, 4-gram, 5-gram ....) along with their frequency ...
2
votes
2answers
48 views

How can I improve feature selection for my Naive Bayes Classifier?

I am classifying companies into two classes ( a particular business type, or not that business type ), using a Naive Bayes Classifier. Specifically, I'm using PHP and PHP NLP Tools. I have two ...
1
vote
0answers
24 views

Best practices to compute TFIDF matrix based on another TFIDF matrix in R

I'd like to compute a TFIDF matrix (tfidf_matrix_b) based on a previously computed TFIDF matrix (tfidf_matrix_a). Is there a ...
1
vote
0answers
21 views

Text mining of machine logs to find correlation between errors in R [duplicate]

I've with me 50 MB data from a machine consisting of event logs such as device status, warning and error. I wish to perform text mining on the same to find correlation between errors i.e. one error ...
-2
votes
1answer
29 views

How to do text clustering for a set of around 10000 messages?

I have around 10000 messages in a variable, i want to form clusters of them based on similarity, so that I can assign some class say 1-10, if 10 clusters are formed and run analysis on them. How can ...
9
votes
2answers
368 views

Bag-of-Words for Text Classification: Why not just use word frequencies instead of TFIDF?

A common approach to text classification is to train a classifier off of a 'bag-of-words'. The user takes the text to be classified and counts the frequencies of the words in each object, followed by ...
1
vote
1answer
113 views

Text analysis : What after term-document matrix?

I am trying to build predictive models from text data. I built document-term matrix from the text data (unigram and bigram) and built different types of models on that (like svm, random forest, ...
0
votes
0answers
26 views

machine learning for a ontology classification problem

I am working on a ontology based classification problem.The main objective was: computing ontology has keywords related to different categories.Each category talks about the domain it is related.For ...
0
votes
0answers
26 views

Probability that a large corpus of text is generated with the same parameters as a subset

Let's say I have a process which generates different words at a set (unknown) frequency per word. I sample this process X times, generating the word "yo" Y times. I then look at a subset of my ...
0
votes
0answers
43 views

How to evaluate and compare two clustering algorithms in R for text mining

I am doing research in R language for text mining. I would like to know how to evaluate and compare two clustering algorithms in R for text mining?
2
votes
2answers
110 views

Supervised keyword extraction: results

I've considered the keyword extraction method as a classification problem (1 = author generated keyword, 0 = no keyword) and I've tried to automatically extract keywords from text document. I've done ...
0
votes
0answers
38 views

How to simulate a multivariate Logistic-Normal distribution in Python

I'm trying to generate a text document using reverse "Correlated Topic Models", which is an advanced version of LDA (Latent Dirichlet Allocation). In this version the topics are generated over a ...
0
votes
0answers
28 views

How do I weight inputs to a regression model so that one figures into the model more than the other?

I have obtained a series of weights from a text mining algorithm. Unfortunately, my algorithm is not capable of doing certain tasks that are too similar without some sort of regression analysis, say ...
1
vote
1answer
68 views

Text mining - Match product title with a description

I have a file with a list of product descriptions. These product description are long strings. Eg "These blue pants are a resistant and comfortable product for tracking and ciclying. In the picture we ...
1
vote
1answer
58 views

Is it necessary to lexicalize the text corpus before applying lda?

While going through a sample lda example code in R R code for topic modelling, the ...
1
vote
0answers
37 views

Difference between Bag of words and Vector space model

I am searching for the intuitive difference between Bag-of-words and vector space model? Is there any relationship exists between bag-of-words and vector space model. I tried searching but couldn't ...
7
votes
1answer
323 views

Automatic keyword extraction: using cosine similarities as features

I've got a document-term matrix $M$, and now I would like to extract keywords for each documents with a supervised learning method (SVM, Naive Bayes, ...). In this model, I already use Tf-idf, Pos ...
0
votes
0answers
53 views

Use Word2Vec to generate a synthetic text dataset

I'm trying to create a realistic set of document like text datasets. Is there any known way to implement the word2vec representation of words in order to manipulate such text?