Tagged Questions

2
votes
1answer
39 views

Polynomial regression using scikit-learn

I am trying to use scikit-learn for polynomial regression. From what I read polynomial regression is a special case of linear regression. I was hopping that maybe one of scikit's generalized linear ...
0
votes
1answer
106 views

Test for linear separability

Is there a way to test linear separability of a two-class dataset in high dimensions? My feature vectors are 40-long. I know I can always run logistic regression experiments and determine hitrate vs ...
1
vote
2answers
264 views

Hybrid (K-means + Hierarchical ) clustering

I have a huge dataset (50,000 2000-dimensional sparse feature vectors). I want to cluster them in to k (unknown)clusters. As hierarchical clustering is very expensive in terms of time complexity ...
0
votes
1answer
502 views

Gaussian Process regression for high dimensional data sets

Just wanted to see if anyone has any experience applying Gaussian process regression (GPR) to high dimensional data sets. I'm looking into some of the various sparse GPR methods (e.g. sparse ...
6
votes
1answer
166 views

What software (paid or free) exists for learning large datasets?

Is there available software (or even just relevant papers) that can perform multiclass learning on datasets of 200m+ samples with 50+ classes and 1000+ features? What are the limits on dataset sizes ...
11
votes
0answers
302 views

First step for big data ($N = 10^{10}$, $p = 2000$)

Suppose you are analyzing a huge data set at the tune of billions of observations per day, where each observation has a couple thousand sparse and possibly redundant numerical and categorial ...
3
votes
3answers
316 views

Divide-and-conquer approach for hierarchical clustering

I have a huge data set (33K), each represented as a bit-vector of 275-dimensions. basically my data set can be represented as a 33000 x 275 matrix. I want to cluster these bit-vectors. I have tried ...
6
votes
1answer
196 views

Dealing with very large time-series datasets

I have access to a very large dataset. The data is from MEG recordings of people listening to musical excerpts, from one of four genres. The data is as follows: 6 Subjects 3 Experimental repetitions ...
6
votes
3answers
176 views

Learning on huge datasets

Basically, there are two common ways to learn against huge datasets (when you're confronted by time/space restrictions): Cheating :) - use just a "manageable" subset for training. The loss of ...
1
vote
2answers
328 views

How to decrease training set size?

I have a large training set, and it is too big to apply some algorithms due to computation limits. What are the common methods to decrease training set size without losing significant amount of ...
6
votes
3answers
1k views

How to quickly select important variables from a very large dataset?

I have a dataset with about 2,000 binary variables/200,000 rows and I'm trying to predict a single binary dependent variable. My chief goal at this stage isn't getting accuracy of prediction, but ...
47
votes
6answers
3k views

What skills are required to perform large scale statistical analyses?

Many statistical jobs ask for experience with large scale data. What are the sorts of statistical and computational skills that would be need for working with large data sets. For example, how about ...
13
votes
5answers
2k views

Free data set for very high dimensional classification

What are the freely available data set for classification with more than 1000 features (or sample points if it contains curves)? There is already a community wiki about free data sets: ...