Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

0
votes
0answers
11 views

Text Classification Issues

In text mining, there are too many variables (features) and the features have multiple permutations. E.g. best, not best, not the best, one of the best, bestest, may not be the best etc. creating ...
0
votes
0answers
6 views

Text Mining - Reports Optimization

I am working on optimizing ~8000 financial and operational reports which have frequency ranging from monthly, quarterly and yearly. To accomplish this I am using text mining to identify similar and ...
-1
votes
0answers
17 views

Automatic mail classification

I'm building a mail classifier in Python 3. I've successfully built classifier to classify spam/ham using SVM (LinearSVC to be precise) using scikit-learn. But the next challenge is to auto bucket the ...
0
votes
0answers
9 views

How do you compare words or documents using LSA (latent semantic analysis)

As the title says, I am a bit cofunsed in how documents or words are compared using LSA (when I say compare, I am referring to calculate similarities, for instance, cosine similarity). In An ...
0
votes
0answers
6 views

Content Based Document classification

I have a corpus of 10 million resumes. I want to add tags to these resumes like Software Engineer, Data Scientists, ...
1
vote
0answers
24 views

calculating PMI for co-occurrences of words

I am in the process of building a question answering system. I am interested in calculating the PMI for words $x$ and $y$ occurring within 5 words of each other in a document. I have the formula and ...
0
votes
0answers
10 views

How do I mine text data for a list of “unspecified keywords” from a bunch of documents?

I need analytics on "skills" from a bunch of documents. The straightforward way for me is to first create a dictionary of "skills keywords" and then get descriptive analytics from the documents i.e. ...
0
votes
1answer
39 views

Text Classification for (Short) Open-Ended Survey Responses?

I am new to text mining/classification, and really want to learn more from this community. My data are open-ended survey responses (n=about 6,000) in which the respondent described what happened at ...
0
votes
0answers
7 views

Comparing two annotated datasets

My task is to compare annotation quality of two datasets, one with respect to another. The datasets have been generated by two distinct groups of people, one is believed to be more ...
3
votes
1answer
24 views

Document classification sample size

I'm working on a document binary classification problem where I have a decent sized corpus of about 30,000 documents (600-1000 words each). My approach is to select a sample of documents and manually ...
3
votes
2answers
55 views

Visualize a lot of words

I have a CSV file with a set of words occurrences in several documents. The first column is the document it. The second column states the text topic (there are 5 different topics). The other columns ...
3
votes
2answers
74 views

Theoretical justification for bag of words

Bag of words and visual bag of words are successful machine learning approaches. Does anyone know of a theoretical justification for why / when they work? What I am trying to explain is the good ...
0
votes
0answers
14 views

kmeans convergence text data

I'm using kmeans to cluster text data. My tfidf matrix is approximately 5700 documents x 3900 features, and sparse as is typical with text data. I have set max iterations to converge = 100. I've ...
0
votes
0answers
11 views

Latent Dirichlet Allocation and text Pre-Processing

I think I understand the basic principles of LDA. However, browsing the githubs of people who applied this method, I noticed they pre-process the Corpus very specifically. For example, about the ...
0
votes
0answers
10 views

Do “bad training instances” decrease predictions quality in multi-label text classification with SGD?

I have 150k company descriptions (~140 characters long) tagged with approximately 1-6 industries. I have 110 possible industries. Industry distribution across different companies is not homogeneous: ...
1
vote
1answer
34 views

Metric for residuals in spherical K-means

I am attempting to use the bag-of-words approach to examine a large text data set. I am experimenting with using spherical K-means to cluster either documents or terms with respect to the other. I ...
0
votes
0answers
14 views

n-gram language model

At the end of the introduction of A Neural Probabilistic Language Model (Bengio et al. 2003), the following example is given: Having seen the sentence ...
1
vote
1answer
35 views

Curse of dimensionality with language models

In the seminal paper A Neural Probabilistic Language Model, Yoshua Bengio and his colleagues make the following point: If one wants to model the joint probability distribution of 10 consecutive ...
0
votes
0answers
11 views

Using document tags to drive prediction

I am working on a project to predict someone's field of study using a selection of tags generated from writing samples. There are between 1 and 50 tags associated with each person, and there are about ...
1
vote
0answers
47 views

Which model for this information extraction problem?

I am trying to solve the following pattern recognition / information extraction problem. Assume I have a text where each token has been annotated by a single class among $K$ classes available (with a ...
0
votes
0answers
25 views

Cluster the tweets using important hashtags

i extract and rank a list of the important hashtags (using td-idf ) from the twitter dataset(twitter.csv) that just includes list of tweets and now i have 9 important hashtags, now i want to use those ...
0
votes
1answer
33 views

How to find similar document with new vocabulary

I am working on a problem of finding similar documents. I am using a Tf-Idf based vector space model representation of documents and it gives me good results. However when I encounter a document ...
1
vote
0answers
38 views

Finding randomly excluded words in hundreds of documents

I have a problem that I am trying to solve using data mining techniques. What is known: There is 253 1 page documents that belong to 4 exclusive topics "clustering" "classification" "frequent ...
0
votes
0answers
13 views

Could i combine character and word n-gram features on a single feature space?

I'm working on a text classification problem using n-gram language models, I have both separate models (character-based and word-based) but I wanna combine them and I'm wondering if that's possible.
0
votes
0answers
11 views

Extact keywords from texts in a corpus for association rule mining

This is my setting, I want to do association rules mining from a corpus. I want to find rules of the form: If A THEN B Where A and B are words. I am using the APRIORI algorithm for computing the ...
0
votes
0answers
21 views

How to assign a document to more than one class in Rapid Miner

I have a sample of 10K call centre text transcripts that have been classified according to a list of standard call reasons (taxonomy) My challenge is that a call can be for more than one reason - so ...
1
vote
0answers
20 views

Reducing the dimension for partial string matching between two large files

I have two very large files, including POIs and have a tool that could do fuzzy matching between each pair of strings between these two files. The problem is that, it is an ...
0
votes
1answer
74 views

Python Text Classification Features Engineering

I am trying to train a model on text classification. I have a large labeled dataset. Documents are set of comments, notes on a incident. Labels are high level categories for the incidents. As ...
0
votes
0answers
13 views

Extracting document's keyword after dimensionality reduction

Let's say I have a word document matrix and I applied SVD to this matrix (LSA), and now I have the representation of this matrix in a reduced dimensional space. How could I use this for extracting ...
0
votes
0answers
41 views

Defining the proximity between sentences

What are the common ways to define proximity between two sentences ? I did various tries, including: Jaccard Index Cosine similarity Levenshtein distance either using the counts for the Jaccard ...
3
votes
1answer
41 views

Statistical tests for 3-gram collocations extraction

I'm trying to extract collocations from some text data, and I use statistical tests to tell if an n-gram is a good collocation candidate or not. All the sources I came across so far (for example, ...
3
votes
1answer
38 views

Is there an NLP or text classification method for word order and pre-specification of terms?

I'm fairly new to NLP and text classification world, and so far I haven't been able to find the answer to the following problem: I have text entries for a large number of observations. My goal is to ...
0
votes
1answer
63 views

Topic modeling (LDA) and n grams

I am new to text mining.... reading stuff about it and setting up some KNIME workflows and Python tooling. I'd like to analyze customer feedback from surveys to derive topics from it without going ...
0
votes
1answer
55 views

Clustering related areas with k-means in WEKA

I am trying to cluster related areas of knowledge. A sample of my file is: ...
0
votes
0answers
10 views

What is the best way to rank paragraphs in relevance to keywords?

I have keywords that describe various paragraphs and those keywords have an ID number. For each keyword, there are multiple paragraphs that are described by it. How would I rank these paragraphs in ...
0
votes
0answers
33 views

How to incorporate cosine similarity into PageRank using SAS/IML?

I want to rank paragraphs based on relevance to keywords and I thought about using cosine similarity to rank thhe paragraphs. How would I incorporate cosine similarity into PageRank? I wanted to use ...
0
votes
1answer
57 views

Linear regression with categorical independent and dependent variables?

Can linear regression be used when both the dependent and independent variable are categorical? i am looking at word-frequency distribution among a series of texts, and want to show that there is a ...
1
vote
1answer
211 views

Cosine angle calculation for the documents - Dissimilarity function not working in tm package in R

I want to find the document similarity. I am using the below R code read 1000 txt articles from directory data/txt corpus <-Corpus(DirSource("data/txt"), readerControl = ...
1
vote
1answer
80 views

outliers in text document clustering

I am using k-means for text categorization. I have some predefined labels (categories) which I want the unlabeled documents to be clustered to. There are some documents that doesn't fit in any of the ...
0
votes
1answer
39 views

Relation between Language Identification and Text Mining?

First of all, sorry for the bad english. The point is: Can i say that Language Identification is part of Text Mining? If yes, what is exact relationship between these two terms?
0
votes
0answers
69 views

Latent Dirichlet Allocation yields different posterior distribution than simple Bayesian model

Method A: out of the box LDA I am using a package to run LDA on a sample of size m with n words in the vocabulary. The end ...
1
vote
1answer
96 views

Use tm_filter to search for multiple words

I´m new to R, so please bear with me. So, I know I can use the following to search for a word in several documents. ...
0
votes
1answer
70 views

How to continuously computate of tf-idf for relevance of single terms

I have a document corpus containing over 4 million documents. Now I want to build an index over terms from the documents of the corpus. Based on the tf-idf of these terms, I want to remove the least ...
1
vote
1answer
110 views

Naive Bayes Classifier in R with class weights

I'm searching for a Naive Bayes classifier in R where I can add a paramter for class weights. I need this, because my data is highly unbalanced. Eg.: Class1: 1000 examples Class2: 800 examples ...
0
votes
0answers
35 views

Meaning of SVD plot of $U$ and $V^T$

I am using SVD/PCA for text mining purposes. Having a $(|terms|,|documents|)$ normalized matrix $M$, by applying SVD, I should be able to reduce the dimensionality and just keep the most meaningful ...
1
vote
1answer
35 views

Is multiple stage binary classification a good idea if you have very few positives?

The problem is the following: We have a set of, say 5000 documents, with a single binary label. Say that 4900 documents are negative and only 100 are positive. I built a binary classifier while ...
2
votes
1answer
57 views

why add one in inverse document frequency

My textbook lists the idf as $log(1+\frac{N}{n_t})$ where $N$: Number of Documents $n_t$: Number of Documents containing term $t$ Wikipedia lists this formula as a smoothed version of the actual ...
1
vote
1answer
75 views

How to use TF-IDF for features selection in Text classification?

I have a small confusion regarding TFIDF. I am planning to use TFIDF for creating better word dictionary to be used in Naive Bayes classifier. I am calculating the TDIDF of all words in respective ...
0
votes
0answers
90 views

Typical range of values for TFIDF

I am working on a text corpus. Each line contains between 10 and 50 words. There are around 25 000 words in the whole text and 1 000 000 lines. I turned this corpus into its tf-idf representation. I ...
0
votes
1answer
601 views

What is VectorSource and VCorpus in 'tm' (Text Mining) package in R

I'm not quite sure what exactly VectorSource and VCorpus are in 'tm' package. The documentation is unclear on these, can anyone make me understand in simple terms?