Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

0
votes
0answers
15 views

Calculating TF-IDF in Matlab for Stop Words

I have been calculating tf-idf for removing stop words in my file. I am experiencing problems. First let us have a look at code I have written.. ...
1
vote
2answers
31 views

How can I analyze my incoming email?

I would like to analyze the email I receive in my Gmail inbox in order to systematically come up with effective Gmail filters for the most common types of email. I am prepared to manually curate and ...
0
votes
0answers
10 views

LexRank damping factor

I am looking into using LexRank to do some text summarization. I am looking at the original paper. One thing that puzzles me is whether a damping factor is used or not. The formulae are all using it, ...
2
votes
1answer
32 views

How can I evaluate the performance of a system that generates word clusters?

The word2vec tool uses deep learning to compute vector representations of words. They've mentioned that - "The word vectors can be also used for deriving word classes from huge data sets. This is ...
0
votes
2answers
25 views

What is the minimum training set size required for a given number of features for document classification?

For document classification problems, is there a rule of thumb for the number of training instances required for the number of terms in the vocabulary? I am using a logistic regression classifier ...
0
votes
0answers
24 views

Compare patterns of text in R

I would like to compare a number of patterns within a huge dataset. I have one variable that is in textform, i.e. 1 der tus wif 2 fiu seg suf erf utz tus wif ...
3
votes
1answer
46 views

Why can we use entropy to measure the quality of a language model?

I am reading the < Foundations of Statistical Natural Language Processing >. It has the following statement about the relationship between information entropy and language model: ...The ...
0
votes
1answer
35 views

How to perform text mining on an online news article?

I am new to data mining and currently working on an online news article from TOI. My aim is to get some useful information out of this text which is not clear when you read the article and the most ...
0
votes
1answer
22 views

Topic and subject classification

I have a set of documents that are OCR-ed and represented as a text file. I want to find out what are the documents that are talking about the same subject and maybe about the same person. I started ...
1
vote
0answers
12 views

How to create character vector of all sentences in a webpage using R [migrated]

I am using R to 'webscrape' a webpage and do text mining on its contents. What I need to do is get a character vector in R where each element in the vector is a sentence from the webpage. Is there a ...
0
votes
1answer
24 views

Data Conversion to Standard data format in hierarchical Dirichlet process

I'm trying to test the performance of posterior inference on a set of documents with hierarchical Dirichlet process for topic modeling. How can i convert my data (document) to standard data format ...
3
votes
1answer
81 views

Machine learning techniques for spam detection, and in general for text classification

I am going to configure a system for spam detection. What I have is a dataset of labeled (spam/not-spam) strings containing, mostly, sentences. I have a background in machine learning techniques, but ...
0
votes
1answer
24 views

how to identify salient words after LSA?

I used Latent Semantic Analysis (LSA) to extract latent topics (i.e., the polynomials coefficient1*word1 + coefficient2*word2 + ...) from a certain corpus. I know that the larger the (absolute value ...
0
votes
0answers
16 views

Posterior distribution for LDA and Newdata

I am using the 'topicmodels' package in R. I tested the posteriori probability for newdata over jss_LDA result by this code : ...
0
votes
1answer
39 views

Automatic labeling of training set

I have once meet the following question, given a training set, is that possible to do the automatic labelling? In addition, if this training set consists of plain text files, is that possible to know ...
0
votes
0answers
24 views

Reviewing classification approach in Weka on Reuters21578

I'm not really familiar with Weka, I learned to use it by watching some tutorials so I am not 100% sure if my approach is the correct one. I have collected the Reuters21758 dataset and I use the ...
-2
votes
1answer
80 views

Document Similarity Gensim

I am trying to get related documents for a list of 10,000 documents from the same set of 10,000 docs. I am using two algorithms for testing: gensim lsi and gensim similarity. Both give terrible ...
1
vote
2answers
70 views

Dataset and papers for baseline [closed]

I'm doing a project about Topic Detection and Tracking in text. I need to perform a baseline so I can compare existing results with mine. I read some papers where they use datasets that are not so ...
1
vote
1answer
22 views

“Exotic” text representations methods?

I'm looking to the different methods of representing a text into a machine-readable format. However, until now, I only found "Bag of Words" approachs with a lot of variations (boolean BoW, weighted ...
0
votes
0answers
30 views

Which diffusion of latent Dirichlet allocation is helpful for assign the words corresponds to each topic?

In my corpus documents I have two different subjects. Which diffusion of LDA (asymmetric or symmetric) could help for assigning the words corresponds to Subject 1 and Subject 2 in my Topics? Here is ...
1
vote
0answers
36 views

How to extract structured information from a text string?

I have a text string containing unstructured data and I would like to analyze it in order to extract structured information. In particular, this text string specifies when a service is operational ...
0
votes
0answers
35 views

Text Books on Text Mining in R [duplicate]

Hi I wanted to know if there are some good books on text mining and classification with some case studies in R .There is similar thread in forum - it didn't serve ...
2
votes
1answer
112 views

Which papers discuss classification or clustering of source code according to programming language?

My specific problem is to separate a huge archive of files containing source code and sometimes including embedded languages (apart from the main language).
2
votes
1answer
47 views

Multi-class text classification with a negative class

I have a multi-class short text classification task with a minor wrinkle: I'd like to also detect when the texts don't fit any of the classes well. I've tried to do it by simply adding unrelated texts ...
0
votes
1answer
52 views

Gensim Topic Modeling

I want to build a standard topic classifier. I was told gensim is the way to do it. I have difficulty training the gensim system. How do we provide a training data in a fast way. Some forum suggested ...
2
votes
2answers
80 views

Selecting a feature modeling approach for text classification

I am new to text processing. Currently I am trying to determine which type of feature vector I need for a classification problem. I am mainly deciding between binary feature modeling and ...
0
votes
0answers
37 views

price prediction

In the project I am working we have a couple of items with different prices. Each item has description but it is possible to find items with similar description. I would like to know how can I ...
1
vote
1answer
61 views

Clarification needed about min/sim hashing + LSH

I have a reasonable understanding of the technique to detect similar documents consisting in first computing their minhash signatures (from their shingles, or n-grams), and then use an LSH-based ...
0
votes
1answer
31 views

Would one frequent term affect the textual document classification quality?

I'm trying to classify twitter messages. For example I collected some tweets about an earthquake and trained a classifier over it. A specific hashtag about the earthquake appears in almost all of the ...
3
votes
1answer
99 views

Keyword clustering

I have one million of keywords (from search queries in google), and I need to group them semantically. I have already done some research and I have found information about how to extract keywords and ...
1
vote
1answer
59 views

Text clustering papers with Precision and Recall

I have created a text clustering algorithm and calculated the Precision and Recall measures for the evaluation. I am looking for papers that contain other text clustering algorithms evaluations with ...
0
votes
1answer
113 views

Calculating pointwise mutual information between two strings

I have a dataset consisting 5000 sentences. I need to calculate PMI between 3-gram and 5-grams in this dataset. For example: The 5-gram is: $x_1$$x_2$$x_3$$x_4$$x_5$ And the 3-gram is: ...
1
vote
0answers
35 views

Calculating and Normalizing ngram relevancy scores from free text extraction

I currently look for a set of ngrams in many sets of documents to establish a relevancy score for each set - eg. I look for the n-gram "adhesive tape" in ~1M sets of 1-500 documents. The values I ...
1
vote
0answers
35 views

What's the methodology behind the most-difference-between-groups-tag-cloud?

What is the likely stats methodology used in this old OKCupid post?: http://www.economist.com/blogs/johnson/2010/10/sexuality_and_language And this: ...
2
votes
2answers
91 views

improve precision in text classification

I am working on binary text classification using sklearn: The length of each sample is not high (~ 200-500 characters) I use TF-IDF to get important words as TfidfVectorizer(sublinear_tf=False, ...
0
votes
0answers
15 views

Typical Dimensions of a Doc Term Matrix

I'm working on a text classification task with about 4.5 million short documents (abt 100-150 words per doc). After performing the standard preprocessing steps (stemming, stripping punctuation and ...
0
votes
0answers
35 views

How to build a relevant text classifier?

I would like to build a message classification system which classifies a given message into either of 2 class - Relevant/Not. I don't have any labelled dataset. I only have certain keywords which ...
0
votes
0answers
12 views

Appropriate accuracy test if drawing distributions on a set of actual values and predicted values?

My question regards statistics as applied to text mining. I have used substring matching to determine the predicted set of keywords. I then classify those keywords into broader groups. Given a ...
0
votes
0answers
34 views

the relationship between training set size and precision/recall

With respect to a classification problem, I once heard the following comment: Adding more positive training cases can increase the recall; and adding more negative training cases can increase the ...
0
votes
0answers
20 views

Evaluate text rarity in document set

I would like to evaluate the rarity of each sentence in a document set. Please let me know the state-of-the-art or a survey paper on this task. I've already checked several papers in the document ...
0
votes
2answers
83 views

A multi-label classification for tagging short text

I am fairly new in the area of text mining and want to practice my skills a little. I have the following task at hand which I want to work on. I have a large list of short texts (~100.000) and every ...
1
vote
0answers
58 views

How to explain difference of importance between feature selection and model quality?

I have a data collection with a mixed feature set consisting of both numerical features and text features. The number of numerical features is quite small, i.e., 6, comparing to the number of text ...
1
vote
1answer
64 views

Classification of data with incomplete label sets

There is such a problem: we have to process multi-label classification (assignmet of tags) of text articles, using some pre-labeled training set. But for many texts in the training set, should be ...
0
votes
0answers
50 views

Best statistical solution for text mining

I'd like to choose a statistical solution for the next years which can manage the major forms of text mining analysis, like mood detection. Is there a solution which stands out from others (R, SAS, ...
0
votes
1answer
162 views

Search in TF-IDF

I want to find the similarity between a document with documents coded as TF-IDF in a pickle file (Python). TF-IDF is done as offline so there is no problem, but when I send a new document for ...
2
votes
2answers
288 views

Why does Naive Bayes outperform Support Vector Machines?

I have a dataset composed of about 36000 attributes and 550 samples, the dataset is generated from text communication between people in some chatrooms. The questions is when I try to classify these ...
2
votes
1answer
66 views

How to determine if short strings of text are closely related to a larger text?

I have 1 short string of text (let's say it's a tweet, max 140 characters): "A review of my beloved Roku 3 media player" I also have a larger body of text (like a ...
1
vote
0answers
75 views

Using Zipf's law to select top K ngrams?

Lets say we extract all the ngrams (unigrams, bigrams, trigrams) along with their frequency from a text. Now if we want to select say some Top K ngrams, is there a way to estimate a good value of K ...
0
votes
1answer
51 views

How to convert numerical values to ML feature in the range [0;1]?

I am supposed to extract a bunch of "generally useful" features from a piece of text. Use cases vary, but one could be text categorization. One thing that springs to mind here of course is the length ...
0
votes
0answers
47 views

Tokenizer for text categorization

In text categorization, it is common to use tokens as features. But, there are several different ways to convert a sentence to tokens. For example, see the different types of Tokenizers here: ...