0
votes
1answer
44 views

How would a decision tree algorithm apply to the Mushroom dataset [on hold]

I'm looking for dataset to explore that would give me an opportunity to understand the pros and cons various ML algorithms; such as decision trees, nearest neighbor, SVM, and neural. I'm going ...
0
votes
0answers
15 views

Removing labelling noise

I have a big data set with unlabelled observations (several million) and about 20 thousand properly labelled ones. There are only two classes and all correctly labelled samples belong to the same ...
2
votes
1answer
27 views

What do NORB and CIFAR stand for?

The MNIST dataset is a standard benchmark data set of digit images. MNIST stands for 'Mixed National Institute of Standards and Technology'. The NORB dataset is a commonly used dataset of binocular ...
0
votes
0answers
50 views

Training and testing on Unbalanced Data Set

I used SMOTE algorithm in R for class balancing. My data size has 13000 rows, I had 7% minority class in my sample now I used SMOTE( Synthetic Minority Oversampling Technique) for class balancing such ...
0
votes
1answer
37 views

Data preparation and machine algo for ad click prediction

I am an ml noob. I have a task at hand of predicting click probability given user information like city, state, os version, os family, device, browser family browser version, city, etc. I have been ...
1
vote
2answers
126 views

How to deal with unbalanced data

I'm doing data analysis with a dataset of 11795 data points (with 88 features). 85% (9973 points) of these data points correspond to data points belonging to class 1, 5% (589 points) belong to class 2 ...
1
vote
1answer
70 views

Original source for the “play tennis” dataset

A famous toy example in machine learning, especially with learning decision trees, is the well known "play tennis" dataset. Is there an official source for the dataset which could be quoted in ...
0
votes
0answers
54 views

How to prove the significance of features in classification?

I have a binary classification problem. I have extracted 500 features from a set of 5000 samples using my domain knowledge. In other words, I have got hand crafted features. I wish to prove that ...
2
votes
2answers
118 views

random forest for large number of variables and predictions

I have very large number of variables compared to samples they are measured on. The following is example data in R. ...
0
votes
0answers
42 views

Should I use multi-label classification?

I have a classification problem with 2 classes (positive and negative). Usually, in such classification problems, all the samples will be labelled either 'positive' or 'negative'. In my dataset, some ...
0
votes
2answers
109 views

Handling unbalanced data using SMOTE - NO BIG DIFFERENCE?

I have a classification problem with 2 classes. I have nearly 5000 samples, each of which is represented as vector with 570 features. The positive class samples are nearly 600. Meaning, I have a 1:8 ...
1
vote
0answers
12 views

Knowledge based annotation vs. Corpus based annotation

What is the difference between knowledge based annotation and corpus based annotation in the context of training a Machine Learning algorithm?
3
votes
2answers
239 views

Improve classification with many categorical variables

I'm working on a dataset with 200 000+ samples and approximately 50 features per sample. Among these 50 features you have 10 continuous variables and the rest of it are categorical variables ...
0
votes
1answer
31 views

how to compute minimum required vc dimension for a classifer to classify a specific data

Suppose we're given an N dimensional data to classify. To cope with this task we may choose a classifier that suits our desires more. However obviously not every classifier is capable of classifying ...
0
votes
0answers
24 views

What is data augmented by the additive inverse?

I am reading Biclustering of expression data (Cheng and Church, 2000) The paper is about the Cheng and Church biclustering algorithm and its main metric, the mean squared residue (MSR). It is said ...
1
vote
0answers
26 views

What sort of feedback button should I implement to measure the difficulty of a task

---background info--- My company has built a Search Engine Optimisation product that can suggest monthly prioritised SEO activities (task suggestions), that will have the most benefit to the end user. ...
2
votes
1answer
55 views

What do I call a set of datasets

For training a Machine learning model, I have 3 datasets: Training Validation Testing Normally I obtain there by dividing up the full dataset into pieces. I've created a function in my code to ...
0
votes
0answers
30 views

How to reduce the dimension of a test data and make it uncorrelated?

I am working on classification of 16000 cell images. Each of them consists of 706 features related to intensity, morphology, colocalisation and texture of the cell. The train set consists of 11 ...
1
vote
1answer
85 views

Difference in meaning of these terms: Dataset vs Corpus

I have a question pertaining to definition of terms that I couldn't find answer for. What's the difference between Dataset and Corpus? I've seen them being used almost interchangeably. My ...
0
votes
1answer
116 views

data normalisation problems

I am pretty new in machine learning and hence facing a lot of confusion in data normalisation concepts. Someone pls clarify the following doubts : 1) While normalising a data matrix of m-samples x ...
0
votes
2answers
151 views

How to use the datasets to practice machine learning algorithms?

I'm a beginner in ML, and I want to practice with some algorithms I learned from the book, after searching stats.SE, I found some places where I can get some datasets. But I have this stupid question, ...
1
vote
0answers
91 views

Organizing data to feed random forests

I'm willing to apply machine learning with R (I will start with random forests then maybe have a look at NNs) on some data, but I don't know where to start, ...
1
vote
1answer
107 views

Machine-learning input data distribution

I'm trying to build a binary 1/0 ML classification algorithm, and was thinking about how to set up the input dataset. If the event I want to predict (the 1's) occur relatively less frequently in the ...
1
vote
1answer
129 views

Entropy in decision tree classification

In the decision tree based classification technique. What is the difference among the different approaches like entropy, gini index? When to use entropy and when to use gini index?.
2
votes
1answer
51 views

How to classify data having sub-instance features?

I am trying to use machine learning on some peculiar (at least for me) data. Usually, when I do machine learning I am use to have the data in this format: ...
-2
votes
1answer
69 views

What are different measures that say something about a given dataset? [closed]

Given a data set (having n instances and m features), what are the different measures that give some insight about some properties of the data set? In other words, if there are two such data sets, ...
0
votes
2answers
87 views

Can I replenish my dataset with unlabeled data?

I have a database of time series data generated by giroscope sensor. A small part of the data is already labeled and used to fit the model. My strong assumption is that most of the samples left in ...
1
vote
0answers
39 views

Discretizing all attributes vs. discretizing only the class label

I have a numeric dataset and I would like to apply classification algorithms on it. So I am using Weka's Discretize filter to convert the numeric values to discrete. However I wonder what can be the ...
2
votes
2answers
209 views

Split dataset randomly

I have a database with 500 records. I want to split these records to 75% and 25% *randomly*in order to use the different datasets for training and testing to machine learning algorithms. Does anyone ...
-1
votes
1answer
93 views

Easy way to test usefulness of data for stock market analysis? [closed]

I have a lot of data (gigs) that may be useful in predicting equity prices. I can import these as a series of features (columns) in a table where the companies are rows. I have time series information ...
1
vote
0answers
77 views

Determining optimal height for regression tree

I have a data set of approximately 400,000 records (for those of you who know, the data set is the one provided by yahoo for their yahoo learning to rank challenge). From this data set I learn a ...
8
votes
1answer
3k views

R vs Python for Data Analysis [duplicate]

Possible Duplicate: Python as a statistics workbench I am just starting out with data analysis and machine learning. From the books that I am reading/have read Python and R seem to be the ...
5
votes
1answer
1k views

Distant supervision: supervised, semi-supervised, or both?

"Distant supervision" is a learning scheme in which a classifier is learned given a weakly labeled training set (training data is labeled automatically based on heuristics / rules). I think that both ...
4
votes
2answers
281 views

Comparing 2 classifiers with unlimited training data

I would like to compare 2 text classifiers C1 and C2, which can be trained with "unlimited" noisy training datasets, meaning that you can use as much data as you want for training, such data being ...
8
votes
2answers
120 views

Learning from relational data

Settings Many algorithms operate on a single relation or table, while many real-world databases store information in multiple tables (Domingos, 2003). Question What types of algorithms learn well ...
4
votes
0answers
487 views

High-dimensional Regression Datasets [closed]

Am looking for pointers to publicly(online) available high-dimensional regression datasets for evaluating my research work. By high-dimensional, am looking for regression datsets with the number of ...
2
votes
2answers
449 views

Baum-Welch training example

I'm the author of a new Baum Welch trainer using MapReduce for the Apache Mahout project (https://issues.apache.org/jira/browse/MAHOUT-627) I'm looking for an example with a reasonably small data set ...
9
votes
3answers
275 views

What is the most efficient way of training data using least memory?

This is my training data: 200,000 Examples x 10,000 Features. So my training data matrix is - 200,000 x 10,000. I managed to save this in a flat file without having memory issues by saving every ...
2
votes
3answers
230 views

What impact does increasing the training data have on the overall system accuracy?

Can someone summarize for me with possible examples, at what situations increasing the training data improves the overall system? When do we detect that adding more training data could possibly ...
5
votes
1answer
436 views

Can I have too much data?

I am training a large set of neural networks for a quite simple task. 10 of the networks have the same configuration, but have different amount of data. The 10 networks each have one hidden layer, ...
4
votes
2answers
260 views

How to classify country names given possible alternate spellings or abbreviations?

Let's say I have a list of users who have specified the country they reside in by typing in something. I want to find the total number of users who came from the US, the UK, and everywhere else. But, ...
6
votes
4answers
1k views

Benchmark dataset for decision tree algorithm

I'm implementing a decision tree algorithm, and I'd like to get a feel for how it performs relative to other implementations. Can anyone recommend popular datasets for training and testing decision ...
13
votes
7answers
2k views

What do statisticians do that can't be automated?

Will software eventually make statisticians obsolete? What is done that can't be programmed into a computer?
3
votes
1answer
112 views

Is it valid to judge the superiority of a classification technique using only artificial data?

... where the significance of the results has been checked with statistical tests. If no, then why? side note: The data is generated based on specified prior probabilities.
0
votes
1answer
395 views

Is there a corpus specifically for categories like sports, entertainment, or health?

I am experimenting with Classification algorithms in ML and am looking for some corpus to train my model to distinguish among the different text categories like sports, weather, technology, football, ...
0
votes
2answers
255 views

Factors that affect variation in the data?

What is the practical way to identify the factors that create variation in a data of a dataset? What category does this question fall into? Are there a set of algorithms that can be used for this ...
4
votes
1answer
1k views

Best practices for measuring and avoiding overfitting?

I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting? I ...
2
votes
3answers
420 views

Learning multiple output

Can you suggest me an algorithm and probably a real code for multiple output learning, where input of the model is vector of around 10 000 values and output is, for each input vector, an output vector ...
2
votes
1answer
194 views

Interpretation of “one” feature change in a supervised classifier

i'm making experiments using app. 5000 labeled dataset.i'm trying different supervised ML algorithm to evaluate the results.The vector size is 13 with the labels (totally 12 features+1 label) and i ...
16
votes
5answers
4k views

Free data set for very high dimensional classification

What are the freely available data set for classification with more than 1000 features (or sample points if it contains curves)? There is already a community wiki about free data sets: ...