Large data are difficult to process and manage because their size are usually bigger than the limits software can normally deal with.
0
votes
0answers
10 views
Reduce Sample Size (Categorise??) [duplicate]
I have been looking for an answer to my question but havn't been able to find any literature documentation or help in regards. I have a large dataset which I need to further analyse. Because of this I ...
3
votes
0answers
30 views
Which variables are driving correlations within groups
I'm running an analysis on a few data sets that each typically have 100-200 cases measured across 120-160 variables - something similar to looking at gene expressions. Each variable is a non-centered ...
2
votes
1answer
44 views
Polynomial regression using scikit-learn
I am trying to use scikit-learn for polynomial regression. From what I read polynomial regression is a special case of linear regression. I was hopping that maybe one of scikit's generalized linear ...
0
votes
0answers
21 views
Large scale k nearest neighbor search
For example we have n samples with vector length k (n>>k).
And we can't load this matrix in RAM at once.
Is there any solutions for large scale nearest neighbor search? any libs suitable for this? ...
0
votes
1answer
50 views
Algorithms for regression analysis which can handle large scale datasets
I am a CS undergraduate student and for my final project i developed a regression algorithm that is suited for large-scale datasets (i wouldn't say 'Big Data', but still large scale).
For the final ...
0
votes
0answers
41 views
CV acc mismatch the prediction
Setting the Context :
My project is in C++, I'm using OpenCV svm here
I used the function train_auto for the CV, however, I implemented my own cross-validation base on this Matlab example here (I ...
3
votes
0answers
55 views
Cluster on high dimensional categorical data (Images with keywords)
We're looking for clues to perform a Cluster Analysis in a DB with +400K images which have keywords associated to them.
Each image could have from 1 to 30 keywords.
Total keywords count is +35K.
...
1
vote
1answer
50 views
Sampling small dataset from large dataset with reference to a given variable
Here is a statistics question which I have been thinking about while working with some of my data. I have a large dataset named "bigbird" (say about a billion rows) and I want to randomly sample a ...
0
votes
2answers
68 views
Effective way to speed-up my mass-univariate problem (i.e. a large set of per-point optimization problems)
I am new to numerical methods, and I have to solve a problem of medical imaging. My background is computer science. I have a naive, general question.
Problem Statement:
I have an extremly large set ...
1
vote
2answers
121 views
Evaluation of k-means output for >3D
I'm implementing the k-means algorithm (in R Map-Reduce) and I wanted to verify if the output I'm getting is close enough to the true centroids of the cluster. This is how I'm verifying with a 2D ...
6
votes
1answer
186 views
How to view large time series data interactively?
I often deal with reasonable sized amount of time series data, 50-200 million doubles with associated time stamps and would like to visualize them dynamically.
Is there existing software to do this ...
0
votes
1answer
110 views
Test for linear separability
Is there a way to test linear separability of a two-class dataset in high dimensions? My feature vectors are 40-long.
I know I can always run logistic regression experiments and determine hitrate vs ...
1
vote
2answers
126 views
Clustering large scale data in fine-grained clusters
I've got large amount of data (e.g. 100K) and I want to cluster them in very fine-grained clusters (e.g. 10K). I look for an appropriate algorithm that uses the similarity function instead of whole ...
0
votes
0answers
69 views
curve fitting with extremely “large” numbers
here is an unconventional but apparently workable idea. wondering if anyone has tried something like it, esp looking for references, examples, or nearby related work.
am working on a mathematical ...
2
votes
0answers
33 views
Sampling from elliptic pde solution in high dimensions
What is known about sampling from solutions to elliptic PDE's in high dimensions, where it is computationally infeasible to construct or store the actual solution?
For example, let $u$ solve the ...
1
vote
1answer
59 views
Large Sample Theory
In large sample theory, I'm told that as $n$ grows larger and larger ($n$ being the number of samples in a dataset) that $\sqrt n(\hat \beta_1-\beta_1)$ gets closer and closer to normal distribution. ...
0
votes
0answers
54 views
Large data set for graph kernel benchmarking [closed]
I am doing a project on graph kernel, and to test the implementation we need a large data set (like graph about 10$^4$-10$^6$ nodes). Does anyone know of such a test which would be appropriate for ...
8
votes
4answers
266 views
Out-of-Core Data Analysis Options
I have been using SAS professionally for close to 5 years now. I have it installed on my laptop and frequently have to analyze datasets with 1,000-2,000 variables and hundreds of thousands of ...
1
vote
1answer
147 views
GLM model is singular
I am trying to fit a GLM model to my data (7 million rows, 153 variables) using R.
More precisely I am using the Revoscaler package, but I suppose my issue would apply to other software as well.
The ...
5
votes
3answers
632 views
Combining results from (GBM or any other) model based on samples from a very large database
How would you combine results of model performed on random samples of a very large dataset?
I need to model a very large database in R (~75 million rows) that can not be loaded directly into memory. ...
2
votes
3answers
212 views
Is it true that in high dimensions, data is easier to separate linearly?
I have often seen the statement that linear separability is more easily achieved in high dimensions, but I don't see why. Is it an empirical fact? An heuristic? Plain nonsense?
1
vote
2answers
265 views
Hybrid (K-means + Hierarchical ) clustering
I have a huge dataset (50,000 2000-dimensional sparse feature vectors). I want to cluster them in to k (unknown)clusters. As hierarchical clustering is very expensive in terms of time complexity ...
17
votes
2answers
692 views
How to visualize an enormous sparse contingency table?
I have two variables: Drug Name (DN) and corresponding Adverse Events (AE), which stand in a many-to-many relation. There are 33,556 drug names and 9,516 adverse events. The sample size is about 5.8 ...
1
vote
1answer
186 views
Dealing with high dimension in principal component analysis
For very extreme high dimensions in PCA, the number of dimensions $p$ is larger than the sample size $N$, does PCA work well or does it work at all? By 'work' I mean
does it work mathematically
If ...
5
votes
1answer
155 views
Comparing rates of binomial outcome response in large datasets
This seems like a trivial question yet my lack of distributed training is leading me towards potentially more confusing answers. Hence I would like to field my question here:
I have data on several ...
3
votes
0answers
123 views
(Non-linear) Transformation of confidence interval for multinomial parameters
I have a certain computational biology problem I wish to model. Say I have a vector $\vec{f}$ that yields $\vec{p}$, of which the explicit form amounts to picking $\vec{p}$ as an eigenvector from an ...
0
votes
1answer
518 views
Gaussian Process regression for high dimensional data sets
Just wanted to see if anyone has any experience applying Gaussian process regression (GPR) to high dimensional data sets. I'm looking into some of the various sparse GPR methods (e.g. sparse ...
6
votes
1answer
167 views
What software (paid or free) exists for learning large datasets?
Is there available software (or even just relevant papers) that can perform multiclass learning on datasets of 200m+ samples with 50+ classes and 1000+ features?
What are the limits on dataset sizes ...
0
votes
0answers
347 views
Why regression of fitted vs. observed or observed vs. fitted values yield different results?
I am performing a regression on a large dataset that is fairly noisy. The line I am running in R is:
lmfit <- lm(predictVariable ~ dataSet[,1:10])
so I have ...
4
votes
1answer
563 views
Time series modeling with high-frequency data
I'm looking for some forecasting advice when dealing with seasonal time series data that has a large number of observations. By "large" I only mean a few thousand --- I'm used to such sizes in Data ...
0
votes
1answer
267 views
Whether to use nonparametric tests to compare two groups when sample size is large but assumptions are violated
I have a data set that has 600 observations divided in two groups. I am going to compare the central tendencies (e.g., the means) of these two groups. However, there are violations of classical ...
2
votes
0answers
98 views
Calculating Regression Coefficients for Very Large Observation Matrices
I am trying to run a regression from a 11300x21500 observation matrix (where there are 11300 observations and 21500 independent variables). However, when I try to implement the usual $(X^T X)^{-1}$ ...
2
votes
0answers
89 views
Generalized Linear Models and Curse of Dimensionality
I was wondering what happens to bias and variance of GLM estimates as dimensionality approaches the number of training data points? Specifically in Linear Regression and Poisson Regression?
I know ...
11
votes
0answers
316 views
First step for big data ($N = 10^{10}$, $p = 2000$)
Suppose you are analyzing a huge data set at the tune of billions of observations per day, where each observation has a couple thousand sparse and possibly redundant numerical and categorial ...
-1
votes
1answer
91 views
High-dimensional dependent binary explanatory variable
I am dealing with a data set containing roughly $n=4000$ binary observations $Y_1, \ldots, Y_n$ with $p=1000$ binary explanatory variables. I suspect that a lot of these explanatory variables are not ...
2
votes
0answers
160 views
Is this approach using the central-limit theorem applicable for reasoning about open source project data?
Definitions
Open Source Project:
In short (and roughly described, only to the purpose of clarification) open source projects allow me to have access to the code of a certain 'program'.
...
1
vote
1answer
388 views
Large data set for testing kernel logistic regression [closed]
Does anyone know of a large data set (upwards of $10^7$ rows, but I'll take $10^5$ as well) that would be appropriate for testing kernel logistic regression? Continuous variables, 2 to 50 independent ...
3
votes
3answers
317 views
Divide-and-conquer approach for hierarchical clustering
I have a huge data set (33K), each represented as a bit-vector of 275-dimensions. basically my data set can be represented as a 33000 x 275 matrix. I want to cluster these bit-vectors. I have tried ...
1
vote
1answer
178 views
Trouble using convert() in RTAQ package
I'm having a little trouble early on using the convert() function in RTAQ to convert .csv taq intraday trade data into an .RData format. I type this:
...
4
votes
1answer
96 views
How to handle changing definitions of regions over time in data?
I just found out that my dataset is a lot messier than I expected and I was wondering if anyone here had some advice. I have sales data that is divided into regions (5 big breaks on a national level ...
16
votes
6answers
604 views
Effect size as the hypothesis for significance testing
Today, at the Cross Validated Journal Club (why weren't you there?), @mbq asked:
Do you think we (modern data scientists) know what significance means? And how it relates to our confidence in our ...
6
votes
1answer
198 views
Dealing with very large time-series datasets
I have access to a very large dataset. The data is from MEG recordings of people listening to musical excerpts, from one of four genres. The data is as follows:
6 Subjects
3 Experimental repetitions ...
6
votes
3answers
177 views
Learning on huge datasets
Basically, there are two common ways to learn against huge datasets (when you're confronted by time/space restrictions):
Cheating :) - use just a "manageable" subset for training. The loss of ...
2
votes
1answer
290 views
How to deal with RAM limitations when working with big datasets in R?
I am currently playing around with the MNIST dataset (http://yann.lecun.com/exdb/mnist/) in R. The training set size is 60000x748 and it seems to drain all my memory even when constructing simple ...
8
votes
3answers
387 views
What are some use of dense matrices in statistics?
OK, I am not a statistician (Not even close). I am a High Performance Computing researcher and I wanted a few test cases for Large (Greater than 5000x5000) Dense Matrices. I had asked here and a few ...
1
vote
2answers
332 views
How to decrease training set size?
I have a large training set, and it is too big to apply some algorithms due to computation limits.
What are the common methods to decrease training set size without losing significant amount of ...
5
votes
2answers
943 views
How to compute an accuracy measure based on RMSE? Is my large dataset normally distributed?
I have several datasets on the order of thousands of points. The values in each dataset are X,Y,Z referring to a coordinate in space. The Z-value represents a difference in elevation at coordinate ...
0
votes
1answer
127 views
High dimensional time series
I'm not sure what words I should look for. I have an under determined dataset of 8000 correlated variables (sales) over 12 months (ie 12 observations for each variable). And I basically want to ...
4
votes
1answer
275 views
Suggestions for multi-dimensional clustering
I am working in a genomics project and I ended up having a huge table with around 800 measurements (cases/rows), around 200 channels (columns/continuous variables) and 5 categories (one categorical ...
4
votes
2answers
212 views
Data structures and libraries for high dimensional text analysis with R
I am going to be using R for text analysis (mostly clustering, classification and some visualization) and was wondering what mechanisms R provides for handling high dimensional, sparse data sets. If I ...
