Tagged Questions

2
votes
1answer
164 views

Simple text classifier: classification taking forever?

I work for a small tech startup, and I want to classify or users into demographics based on the domain of their email address. When users sign up to our site, they can enter a job category, or pick ...
0
votes
0answers
56 views

LIBSVM-based classifier assign very low score to positive validation files

Recently, I have been applying the LIBSVM to build a classifier based on a set of documents. The positive set has about 20000 files and negative set has about 50000 files. The built classifier is then ...
0
votes
0answers
17 views

Algorithms where knowing the avg. word length and sentence length in corpus are useful?

A co-worker and I were discussing whether we wanted to find the median word length and sentence length in our corpus or if it was overkill ( my current use case is to make sure that I did not send in ...
1
vote
1answer
66 views

Relationship between number of training set and classification performance

Are there any research/paper on the relationship between the number of documents for training and the classification performance using support vector machine?
0
votes
2answers
82 views

Calculating distance metrics between a sample set and a point

i have a list of text files and i know that these texts belong to a group, by using this group of text files (i.e this is my sample set) i'd like to calculate Jaccard index and Edit distance for each ...
15
votes
1answer
207 views

I want to build a crime index and political instability index based in news stories

I have this side project where I crawl the local news websites in my country and want to build a crime index and political instability index. I have already covered the information retrieval part of ...
0
votes
1answer
92 views

bag of words in an online configuration, for classification / clustering

I have a set of image documents. I extract text keywords from this images using OCR to represent each image as a bag of words (a vector where each value is the number of occurrence of a word in the ...
1
vote
0answers
44 views

Which vocabulary to use for a special version of the Rocchio classification algorithm?

My issue is a bit hard to explain in this question's title, so hopefully I can make clear what my problem is about in this text. I'm dealing with partially supervised text classification. I have a ...
2
votes
1answer
94 views

How to do feature selection for learning from positive and unlabeled examples?

I have a binary classification task for German webpages for which I only have positive examples. That is why I use learning from positive and unlabeled examples as described on this page, also known ...
2
votes
0answers
62 views

Hierarchical decomposition of an imbalanced multiclass classification problem

I have a heavily imbalanced multiclass text classification problem: one class is very probable a priori (P), while the remaining four ones are about equally ...
0
votes
0answers
74 views

Adding training examples to Bayesian classifier reduces accuracy

I'm working on a problem to predict/classify overall sentiment of a large amount of text, which I can verify on the next day. Each data point is a day and is composed of multiple articles. I bin the ...
5
votes
5answers
205 views

How to do one-class text classification?

I have to deal with a text classification problem. A web crawler crawls webpages of a certain domain and for each webpage I want to find out whether it belongs to only one specific class or not. That ...
0
votes
0answers
33 views

Conditional Independence between Feature Sets

I've two feature sets for document classification, Words and Part-of-Speech tags. Given a class label, I'm assuming the conditional independence between them is lacking because the occurrence of ...
1
vote
1answer
99 views

Possible reason for failing to build a support vector machine

I was trying to build a classifier for a set of documents using a support vector machine. I choose to build the feature space using term occurrence. While experimenting, I found the following ...
4
votes
2answers
2k views

Difference between naive Bayes & multinomial naive Bayes

I've dealt with Naive Bayes classifier before. I've been reading about Multinomial Naive Bayes lately. Also Posterior Probability = (Prior * Likelihood)/(Evidence). The only prime difference (while ...
3
votes
1answer
85 views

How to construct an optimally balanced reference corpus for text categorization?

I have a rather large collection of text documents categorized into about 150 categories. While some categories are represented by several thousands of documents, others have only a few hundreds ...
2
votes
1answer
60 views

Classification algorithm and big text field

My question is straightforward: I want to do statistical classification in a database, mostly of floats, but one of the fields of the data is a big text. By big I mean up to 1000 words. What is the ...
1
vote
1answer
250 views

Comprehensive dataset for documents classification

In document classification (document categorization) field, researchers recommend always a few standard dataset, such as Reuters-21578, RCV1(Reuters Corpus Volume 1). However, these datasets only ...
4
votes
5answers
3k views

Is cosine similarity a classification or a clustering technique?

In document classification, is cosine similarity considered a classification or a clustering technique? But you need training data with the cosine similarity for creation of the centroid right?
1
vote
1answer
388 views

What are the differences between document classification and clustering when working with a single topic?

I am doing some web page clustering work and I'm going to use cosine similarity as my distance measure. Even though cosine similarity is a clustering technique, I have to give training data in order ...
4
votes
2answers
158 views

How to classify country names given possible alternate spellings or abbreviations?

Let's say I have a list of users who have specified the country they reside in by typing in something. I want to find the total number of users who came from the US, the UK, and everywhere else. But, ...
5
votes
2answers
511 views

Algorithms for clustering documents by similar words and phrases

I'm working on a project where I'm trying to take a pair of documents and find and group (cluster) similar words and phrases between them. Which algorithm would solve this kind of a problem? I know ...
2
votes
1answer
615 views

Is my interpretation of Neural Network results correct?

I use a neural network with a topology of 17-30-1 (sigmoid, atan error function, mse as cost function, 5-fold cv) for text classification. (It's closely related to a previous question of mine.) The ...
6
votes
1answer
61 views

Classification of conversations based on content

I'd like to be able to design a classifier that can distinguish between different types of conversations (not necessarily tell anything about mood, sincerity, or outcome, that's a bit too far ...
4
votes
0answers
404 views

Document classification with Bayes

I want to build a document classifier in R, using the Naive Bayes approach. Here are steps, that I've done so far: I have corpus with about 30 documents from 2 authors (Classes are: "target author" ...
5
votes
2answers
207 views

Sophisticated models for classifying short pieces of texts

I have about 30000 book names assigned to 6 categories, and I want to build scalable and accurate classifiers. So far I have only been able to use Naive Baye's and LibLINEAR classifiers and they both ...
0
votes
1answer
220 views

Is there a corpus specifically for categories like sports, entertainment, or health?

I am experimenting with Classification algorithms in ML and am looking for some corpus to train my model to distinguish among the different text categories like sports, weather, technology, football, ...
1
vote
1answer
339 views

Obtaining final classifier in RTextTools

I have a question concerning the RTextTools package. When I was reading textbooks and wikipedia I imagined that text classification is all about defining a classificator. In RTextTools you define ...
10
votes
1answer
430 views

Is cross validation a proper substitute for validation set?

In text classification, I have a training set with about 800 samples, and a test set with about 150 samples. The test set has never been used, and waiting to be used until the end. I am using the ...
5
votes
2answers
605 views

Why does ridge regression classifier work quite well for text classification?

During an experiment for text classification, I found ridge classifier generating results that constantly top the tests among those classifiers that are more commonly mentioned and applied for text ...
4
votes
2answers
627 views

How to plot results from text mining (e.g. classification or clustering)?

In text classification and clustering, the number of features are normally big, e.g. I currently get are around 5,000 features which is already really small compared to many other text mining tasks. ...
3
votes
0answers
113 views

Category selection for text classification

It is said that to achieve a good result (many different metrics) for text classification, it is not always a business of selecting the algorithm/classifier. Sometimes, it is even more important to ...
1
vote
1answer
1k views

Text classification with RapidMiner

I have a folder for each class that contains some text files related to that class and need to implement a classifier for these files in RapidMiner with KNN algorithm. Is there any sample?
2
votes
0answers
59 views

Statistical analysis on categories before text classification

I want to classify text by different topics. However, one of the current problems is that there are several topics/categories that are quite intuitively independent and statistically standalone, but ...
5
votes
2answers
958 views

Semi-supervised learning, active learning and deep learning for classification

Final edit with all resources updated: For a project, I am applying machine learning algorithms for classification. Challenge: Quite limited labeled data and much more unlabeled data. Goals: ...
1
vote
3answers
343 views

Text categorization/classification for small scale text

I'm looking into a way to classify/categorize sentences into pre-defined categories (around 10-15). Yes, indeed sentences, not articles or paragraphs. Given the average length of articles are not too ...
8
votes
5answers
2k views

Large scale text classification

I am looking to do classification on my text data. I have 300 classes, 200 training documents per class (so ...
1
vote
1answer
977 views

Text Classification in R [closed]

New to R, and am trying to do text classification. I am using R package tm to convert raw txt data into matrix. Here's the relevant code snippet. ...
1
vote
3answers
239 views

How does a classifier handle unseen documents that do not belong to any of the pre-existing classes?

I am doing text classification, and have been playing around with different classifiers. However I have a pretty basic question: what if a new unseen document comes in and it happens to not belong to ...
5
votes
3answers
270 views

Semantic distance between excerpts of text

I'm wondering how far along the natural language processing is in determining the semantic distance between two excerpts of text. For instance, consider the following phrases Early today, I got up ...
22
votes
6answers
878 views

Statistical classification of text

I'm a programmer without statistical background, and I'm currently looking at different classification methods for a large number of different documents that I want to classify into pre-defined ...