'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

1
vote
1answer
36 views

Comparing large categorical data sets with low or zero counts

I'm dealing with a biological feature which can be classified into $2^{20}$ categories. I also have two pretty large data sets of 1 and 3 million entries. Actually, only around 30 thousand categories ...
4
votes
2answers
93 views

How to reduce the dimension of $10^8$ vectors

I have $10^8$ vectors in $1000$ dimensions each. I would like to drastically reduce their dimensions. However PCA seems computationally infeasible. Are there near linear time methods to do ...
1
vote
1answer
51 views

Best metric for evaluation of mixture-of-Gaussian clusters on big-data

I have made a new algorithm that is specifically crafted for clustering very large datasets. In order to document it as a research paper, I have to choose one or two internal (no-label) cluster ...
1
vote
0answers
11 views

Sybil detection metric

I have a set of user data and I want to build some kind of metric to evaluate the probability of the user being a sybil (a "fake" account). But I have a very limited set of users who are sybils with ...
2
votes
1answer
21 views

Need reasons or references on small p-values with large data sets

What are reasons or references for a statement such as "When you have a lot of data the statistical problem you run into is that even a tiny difference will be statistically significant." I see this ...
0
votes
1answer
36 views

Random forest ML algorithm suitable for use on cluster based HPC?

I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve, the model contains over 12gb of data, including over 500 parameters. Now running small ...
1
vote
1answer
34 views

Testing for heteroscedasticity with many observations

I used the Breusch-Pagan test for heteroscedasticity, but I have many observations ($\approx 500,\! 000$) and the Breusch-Pagan test uses $nR^2$ as a test statistic where $n$ is the number of ...
0
votes
1answer
20 views

Literature for Cross Validation on Sparse Data?

I've read a lot about Cross Validation to estimate prediction error, specifically for selecting the number of components in a PCA model (I'm not doing SVD/PCA, but it's very similar), but I can't find ...
1
vote
0answers
38 views

regression algorithms which work with sparse categorical predictors

I am working with a very sparse problem with a large number of categories per feature and I am currently looking for existing machine learning regression algorithm implementations which can either ...
0
votes
1answer
30 views

Memory consumption in NLP tasks [closed]

I am trying to apply a simple Naive Bayes or SVM (libSVM) algorithms to a large data set, which I've constructed as an .arff file. The number of features in my set is ~180k and there are ~6k ...
0
votes
1answer
54 views

Can I subsample a large dataset at every MCMC iteration?

I have a large dataset from which I want to perform a bayesian probit regression using Gibbs sampling 1. Since the dataset has one milion rows, and variables from a truncated normal must be sampled ...
1
vote
0answers
21 views

Window models in stream data processing

Reading about data stream clustering I met the next terms: landmark window model, sliding window model, damped window. As to sliding window it's clear - oldest data escape the scope, the new data ...
1
vote
1answer
53 views

Comparing two distributions

I have more than five hundred thousands samples of a continuous variable measured in two groups: a treatment and a control one. I would like to decide whether the measurements follow the same ...
-1
votes
1answer
71 views

Clustering large movie dataset using k-medoids?

I have to cluster a movie dataset of 10000 movies. A movie has attributes like Genres, Actors, Directors, Year. Earlier I thought that we can use a simple clustering algorithm like k-medoids and the ...
2
votes
0answers
86 views

Is Big Data the end of Statistics? [closed]

Statistics emerged as a discipline whose goal was to allow inferences about inaccessible populations from data obtained from its samples. Does modern access to huge amounts of data depreciate the ...
2
votes
0answers
31 views

Dealing with Big Data and Lots of Variables

What is a good technique to use on data that has many categorical variables with many possible values? For example, let's say you are trying to determine what kind of people are more likely to ...
0
votes
2answers
82 views

Streaming k-means

I want to perform something like streaming/online/out-of-core kmeans clustering on large data. Here is simple idea: Break all data into N chunks. Read from disk 1st chunk and calculate centroids ...
2
votes
0answers
66 views

Big Data? Have we solved Small Data yet? [closed]

There has been a lot of attention on Big Data recently, where the problems are often more logistical (how to deal with large volumes of data) rather than statistical. At the other end of the spectrum ...
5
votes
3answers
790 views

Is visual inspection the only way to compare large datasets?

I have two large data sets, in fact, one of them is even much larger than the other. Visually, there doesn't seem to be that much difference between them: The actual data underlying the box plot ...
1
vote
1answer
49 views

Alternative to spherical K-Means for clustering large high dimensional dataset

What are some alternatives to Spherical K-Means for clustering very large datasets of high dimension? I'm looking for something that will be fast even on large datasets, and preferably will not ...
4
votes
1answer
291 views

Testing large dataset for normality - how and is it reliable?

I'm examining a part of my dataset containing 46840 double values ranging from 1 to 1690 grouped in two groups. In order to analyze the differences between these groups I started by examining the ...
2
votes
0answers
56 views

Kolmogorov-Smirnov test for large sample [duplicate]

I am investigating if an exponential distribution is a good fit for a large sample of data (200) I have. I have already looked at a histogram but was wanting to investigate further. I was going to use ...
0
votes
0answers
23 views

Clustering algorithm for my situation?

Here is my situation. I have a corpus of over 500,000 news. Now I need to cluster the news based on closeness in time and cosine similarity, using vector-space model and TF-IDF weights. I want to ...
1
vote
1answer
65 views

Impact of data dimensionality on computation complexity of SVM?

What is the impact of data dimensionality on computation complexity of SVM? I found on the literature that the complexity of SVM is $O(N^3)$, where $N$ is the number of training examples. If the ...
0
votes
0answers
14 views

Are there any benchmarks comparing Mahout with older software?

I want to know about Mahout's implementation of classification algorithms - how does its accuracy compare with other large data software? Anyone knows a reference on this? I know about this SAS ...
1
vote
2answers
116 views

Predict only the first N principal components in a PCA analysis

I'm using R to analyze a very large dataset. I conduct a PCA on one dataset, PCA <- prcomp(formula = ~., data = train, scale = T, na.action=na.exclude) and ...
2
votes
2answers
254 views

How to summarize and understand the reults of DBSCAN clustering on big data?

Many clustering algorithms can be used with big data, eg. versions of KMeans, DBSCAN based on Hadoop, etc. But, with k means we will get k centroids for k clusters and we can map them to the space and ...
4
votes
2answers
372 views

What tools do Machine Learning experts use in the real world?

I'm currently taking a class covering some topics in machine learning. The class is taught in MATLAB using Liblinear so far. I was curious though what kind of tools people used in the real world to ...
0
votes
0answers
22 views

Large scale 1-Dimensional Gaussian Process Classification

I have a dataset with a small number of input points (e.g. +- 300), but millions of boolean outcomes. ...
8
votes
2answers
485 views

Why would a statistical model overfit if given a huge data set?

My current project may require me to build a model to predict the behavior of a certain group of people. the training data set contains only 6 variables (id is only for identification purposes): ...
0
votes
1answer
70 views

Sparse PCA/Dictionary learning when the features are extremely sparse?

I am trying to do sparse PCA/dictionary learning, that is decompose a matrix $X\approx UV$ where the loading matrix $V$ is sparse, usually enforced with an $\ell_1$ penalty (the difference between ...
4
votes
0answers
47 views

Splitting a variable with nominal and numeric values

I have a variable that has both numeric and nominal components. The source has a documentation which helps in identifying which is which and for splitting into their proper components. I will do ...
0
votes
3answers
83 views

Statistical comparisons between large data sets

Currently, I am looking for the correct (or suitable) statistical method to compare 4 very large datasets (n = 31 million each), that are based on an experiment where a continuous variable was ...
1
vote
0answers
77 views

What is a good and efficient algorithm for a content based recommender?

I want to build a content based recommender in a restricted environment regarding cpu power and memory (to be specific: a mobile device, but it is not acceptable to build the recommender on a remote ...
0
votes
1answer
41 views

Question regarding significance level [duplicate]

Say, for instance, I'm estimating a model with about 5 million observations using linear regression or MLE. Given that the estimates are consistent, using the standard rule of rejecting the null on a ...
1
vote
1answer
272 views

What data structure to use for my cluster analysis or what cluster analysis to use for my data?

I have a large dataset of categorical variables. The data consists of shoppers who purchased two items during a single trip to a store. There are approximately 75,000 cases and 1,500 different ...
6
votes
1answer
169 views

Star Coordinates vs. principal component analysis

I currently preparing a presentation for a university course in "Visual Data analysis". And one of my topics is the "Star Coordinate" visualization. Star Coordinates As Star Coordinates perform a ...
1
vote
1answer
109 views

Recursive logistic regression merge

I need to make regression on big amount of data, each row have around 1000 features. Did will outcome will be same or better when i make 4 separate regressions of 250 features and after that i will ...
4
votes
4answers
224 views

How do I deal with large data similarity computation?

I have lot of records like this: M is about 10 million and N is about 100K. Now I want to apply collaborative filtering on ...
1
vote
3answers
107 views

Large data analysis - learning resources

My question is very simple: which learning resources (books, courses, online courses, and so on) about "large data analysis" would you suggest to a graduate with a strong background in Machine ...
0
votes
2answers
234 views

Statistical Significance with large data sets [closed]

When I was a Ph.D. student I was trained in no uncertain terms that When we had large numbers of data results, that the number of significant results HAD to be, in and of themselves SIGNIFICANT! I ...
2
votes
1answer
255 views

Non-normality of residuals in linear regression of very large sample in SPSS

I have a dataset of ~17,000 cases in SPSS 21 with which I am trying to run multiple linear regression. I have plotted the Studentised residuals against the unstandardised predicted values and also ...
0
votes
1answer
173 views

Multinomial logistic regression for big data

How do I go about doing a multinomial logistic regression when I have 70 million observations? Is it feasible? It seems that R is out of the question due to memory constraints?
2
votes
1answer
1k views

Mann-Whitney U test with very large sample size?

I'm doing a Mann-Whitney U test to compare two very large samples (sample size 1 = 13250; samlple size 2 = 38871) originating from a raster image. I know t-tests are not recommended to compare ...
4
votes
2answers
380 views

Comparing nested binary logistic regression models when $n$ is large

To better ask my question, I have provided some of the outputs from both a 16 variable model (fit) and a 17 variable model (...
1
vote
2answers
166 views

ANOVA with huge dataset - use only the mean for each condition?

I need advice about how to carry out an ANOVA. I studied some theory of ANOVA, but apparently it is not enough. Basically I collected around 300 reaction times for 12 subjects in my experiment. For ...
1
vote
1answer
79 views

What is a high dimensional multivariate data set?

I'm new to data analysis and data mining. Often in the papers I'm reading, they use the term "high dimensional multivariate data set." Currently, my task is to detect an outlier and visualize the same ...
0
votes
1answer
63 views

Multidimensional scaling for big dissimilarity matrix

I have a large symmetrical dissimilarity matrix of dimension 300 000. Can you please suggest the multidimensional scaling algorithms that can work with such large data? Input of course can be the ...
3
votes
1answer
252 views

What is the time complexity of Lasso regression

What is the asymptotic time complexity of Lasso regression as either the number of rows or columns grows?
9
votes
4answers
227 views

Goodness-of-fit for very large sample sizes

I collect very large samples (>1,000,000) of categorical data each day and want to see the data looks "significantly" different between days to detect errors in data collection. I thought using a ...