Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...
1
vote
1answer
35 views
How to plot a “differential word cloud” in R?
I have a questionnaire with one "open" text box, and 20,000 responses. I'm not planning to read all of them, but I'm thinking of interesting things to do with them. A general tag cloud would not be ...
0
votes
1answer
46 views
How to adjust machine learning training data set with time
I'm using machine learning to do text classification right now, I first use a training data to train my classifier, then use this classifier to classify text document into different classes. With the ...
0
votes
0answers
28 views
Text Rank Algorithm to find Keywords
I was trying to implement the text rank algorithm mentioned in :
http://acl.ldc.upenn.edu/acl2004/emnlp/pdf/Mihalcea.pdf
It seems to be simple to implement in python but I could not get the exact ...
0
votes
0answers
20 views
Frequency of Math Symbols [migrated]
Does anyone know of a study that has calculated the frequency of math symbols based on some popular mathematics journals or math corpus?
For example in English you have letter frequencies of the most ...
1
vote
0answers
20 views
An up to date keyword set on global news
My question is not strictly binds to the topic of text mining, but maybe you can help.
I am hunting for a keyword set, which has the following criterions:
- contains only english words/n-gramms or ...
0
votes
0answers
7 views
Problem with generating document term matrix with 'tm' package in R using Russian language documents [migrated]
I'm having some problems with the resulting Document Term Matrix in R. My documents are in Russian and encoded in UTF-8, but the DTM, when produced, represents the characters incorrectly.
This is my ...
3
votes
0answers
38 views
Latent Semantic Analysis - Co-occurrence of words
Let $A[n\times m]$ represents the term-document matrix, where, $n$ is the number of terms and $m$ is the number of documents. This matrix can be composed into 3 matrices (SVD decomposition) such as,
...
1
vote
1answer
95 views
Loop over Tokens in RapidMiner's Text Processing Plugin
is there any possibility to iterate over the tokens of a text document within RapidMiner? My first try was to window the document after tokenisation. But this seems very complicated.
I'm doing this ...
2
votes
2answers
85 views
Medical Insurance Fraud Detection: Text analysis
I'm trying to analyse a dataset to detect fraudulent insurance claims. Unfortunately, other than basic demographics the rest of the claim is a free format OCR scanned text file made from documents ...
1
vote
0answers
48 views
How to treat sentences including “and so on” when analyzing open-ended responses? [closed]
I'm analyzing responses from open-ended questions, by item (specifically, sections of an entire answer).
I were asked to mention features or actions about a subject. I have sections that I reach to ...
2
votes
1answer
35 views
How to extract ngrams from ambigous text after lemmatization?
After lemmatization of text I have a sequence of sets of lemmas, because every word can correspond to more than one lemma. How should I extract ngram statistics based on that? The only thing that ...
0
votes
1answer
125 views
understanding of libsvm output
I applied libsvm to build a text classifier. The output looks like as follows:
...
0
votes
0answers
85 views
N-grams or vector space model for text mining?
I have a database that contains 5 columns that contain 0 or 1 base on true or false. Each user can choose 4 out of 5 columns which means we can have a max 5 number of 1 and then last will be 0. The ...
1
vote
1answer
61 views
Derivation of the posterior over topics in LDA
When studying the latent Dirichlet allocation, I am not very clear about some procedures in their deriving equations. Please refer to the attached figure, how to understand those two steps, marked as ...
1
vote
0answers
15 views
Statistically fuzzy version of a checksum for file text signature
Background:
Often I end up downloading the same pdf article twice since I do not remember I've already downloaded it. One way around is to maintain an index of cheksums (say md5 etc.) based on the ...
1
vote
0answers
51 views
Choosing a multiclass strategy
What would be the best approach to choosing the multi-class strategy (MSVM, OVA, ECOC) for classifying a large number of classes with limited examples of each class?
Another two factors that define ...
2
votes
1answer
164 views
Simple text classifier: classification taking forever?
I work for a small tech startup, and I want to classify or users into demographics based on the domain of their email address. When users sign up to our site, they can enter a job category, or pick ...
0
votes
0answers
56 views
LIBSVM-based classifier assign very low score to positive validation files
Recently, I have been applying the LIBSVM to build a classifier based on a set of documents. The positive set has about 20000 files and negative set has about 50000 files. The built classifier is then ...
0
votes
0answers
17 views
Algorithms where knowing the avg. word length and sentence length in corpus are useful?
A co-worker and I were discussing whether we wanted to find the median word length and sentence length in our corpus or if it was overkill ( my current use case is to make sure that I did not send in ...
1
vote
0answers
114 views
Code for evaluating perplexity of an LDA topic model
I'm working on a research project where we are trying to assess model fit improvement of our LDA model (the basic Blie et al., 2003 LDA approach). I'm looking for code that allows us to assess the ...
2
votes
0answers
41 views
Measuring dispersion of tokens within a text file
I am working on a code analyzer application. It is essentially a piece of software that parses and interprets other programs' code and comes up with various metrics, findings, statistics, and ...
1
vote
1answer
146 views
How does scikit-learn perform $\chi^2$ feature selection on non-categorical features?
I'm experimenting with $\chi^2$ feature selection for some text classification tasks. I understand that $\chi^2$ test checks the dependencies B/T two categorical variables, so if we perform $\chi^2$ ...
0
votes
0answers
53 views
the effects of feature matrix format on the training time of LIBSVM
I am using Libsvm to perform text classification tasks. I normally uses binary occurrence, TF/IDF to build feature set for the input documents. It normally takes quite longer for Libsvm to finish ...
0
votes
0answers
30 views
Evaluating a text analytics engine for document concepts identification
By document concept identification I mean the problem in which an engine has to detect various concepts mentioned in a document e.g. football, London, medicine etc.
How in general a text analytics ...
1
vote
1answer
73 views
Predicting continuous variables from text features
I want to predict a continuous variable from text features. Lets say I have some student essays and I want to predict their quality, as measured by a human grader, using text features (mostly words ...
1
vote
1answer
66 views
Relationship between number of training set and classification performance
Are there any research/paper on the relationship between the number of documents for training and the classification performance using support vector machine?
0
votes
0answers
21 views
Massive Textual Corpus [duplicate]
Possible Duplicate:
Where to find a large text corpus?
I know someone has asked a similar question here, but I'm wondering whether anyone knows of a large textual corpus that is available ...
0
votes
0answers
59 views
Use of text mining document classification via decision rules?
Edit: could someone please explain decision rules in text mining and what they are useful for and how to obtain them from text?
0
votes
0answers
234 views
Text mining with RapidMiner
I have a codified text written in 3 different codes, each text in a labelled file, the labels are common for the 3 different codes, and I'm trying to input all the texts into RapidMiner for finding ...
0
votes
2answers
82 views
Calculating distance metrics between a sample set and a point
i have a list of text files and i know that these texts belong to a group, by using this group of text files (i.e this is my sample set) i'd like to calculate Jaccard index and Edit distance for each ...
2
votes
1answer
162 views
Chi-squared test for detecting trending terms
I'm trying to find bursty(trending) terms in a text stream. There are two frame in a stream; expected frame and observed frame.
For each frame, I tokenize documents(tweets/blog posts etc.) up into ...
1
vote
0answers
140 views
Calibrating multiple binary SVM classifiers for one-vs-all multi-class classification
I'm classifying text using the one-vs-all approach. There are three classes. I've trained 3 different binary SVM classifiers using 10-fold cross-validation. The accuracy of the binary classifiers ...
15
votes
1answer
206 views
I want to build a crime index and political instability index based in news stories
I have this side project where I crawl the local news websites in my country and want to build a crime index and political instability index.
I have already covered the information retrieval part of ...
3
votes
1answer
212 views
How to optimize hyper-parameters in LDA?
After reading Hanna Wallach's paper Rethinking LDA: Why Priors Matter, I want to add hyper-parameter optimization to my own implementation of LDA. However, the paper doesn't given any details about ...
0
votes
3answers
81 views
Is there any dataset or api that gives a list of infrequent words? [closed]
I'm actually working on an information retrieval project, and I want to extract words that are of significance from a sentence. This is somewhat opposite to stopwords.
In a sentence like: He was a ...
0
votes
1answer
91 views
bag of words in an online configuration, for classification / clustering
I have a set of image documents. I extract text keywords from this images using OCR to represent each image as a bag of words (a vector where each value is the number of occurrence of a word in the ...
1
vote
0answers
44 views
Which vocabulary to use for a special version of the Rocchio classification algorithm?
My issue is a bit hard to explain in this question's title, so hopefully I can make clear what my problem is about in this text.
I'm dealing with partially supervised text classification. I have a ...
0
votes
1answer
83 views
Merge text collection subsamples for cross-validation
I'm doing a research and to test our methods we're using 5-fold cross-validation. Our collections consist of some pre-classified text. To use the data in Weka, I'm pre-processing it with the ...
0
votes
1answer
191 views
Continually updating naive Bayes classifier
I am attempting to use a Naive Bayes classifier to classify text. To accomplish this I have created an Excel sheet with a binary distribution for three variables. The workbook can be found here. ...
3
votes
1answer
89 views
Coding single word responses into consistent
this might be a stupid question, but I'll try anyway!
I have a data set from a survey asking about what brands they can remember within the category toys.
The survey participants get to write a ...
0
votes
2answers
116 views
Text mining discrete paragraphs
I have a spreadsheet with Titles in the first column, descriptions in the second column and whether or not they were included in our screening in the third column. We are attempting to find out which ...
2
votes
1answer
93 views
How to do feature selection for learning from positive and unlabeled examples?
I have a binary classification task for German webpages for which I only have positive examples. That is why I use learning from positive and unlabeled examples as described on this page, also known ...
0
votes
1answer
92 views
Regarding the feature generation method with SVM-based classification method
When using SVM to build classifier for a collection of documents, we can use term occurrence, term frequency or even TF/IDF. I would like to know what are the main disadvantages of using term ...
2
votes
0answers
62 views
Hierarchical decomposition of an imbalanced multiclass classification problem
I have a heavily imbalanced multiclass text classification problem: one class is very probable a priori (P), while the remaining four ones are about equally ...
1
vote
1answer
107 views
The general approaches for improving a SVM-based classifier which is low precision and high recall
I built a SVM-based classifier against a data set, the precision is about 66% and the recall is about 88%. Generally, what are the options to tune the parameter that can increase the precision?
0
votes
0answers
73 views
Adding training examples to Bayesian classifier reduces accuracy
I'm working on a problem to predict/classify overall sentiment of a large amount of text, which I can verify on the next day. Each data point is a day and is composed of multiple articles. I bin the ...
5
votes
5answers
204 views
How to do one-class text classification?
I have to deal with a text classification problem. A web crawler crawls webpages of a certain domain and for each webpage I want to find out whether it belongs to only one specific class or not. That ...
1
vote
3answers
486 views
Good books on text mining?
Hi I wanted to know if there are some good books on text mining and classification with some case studies?. If not some papers/journals accessible to public would do. If they illustrate their examples ...
0
votes
1answer
144 views
RTextTools for coded sentences
I'm new to text mining and I'm not sure if could be applied here:
I have labeled sentences but the words I work with are codes, so each word length ranges from 5 to 15 letters and there are only 4 ...
2
votes
3answers
362 views
Significance test for large sample sizes
This is more of a theoretical question. Super large sample sizes will almost always show a significance when a $\chi^2$ test is done. Is there any other statistical test of significance (an ...
