'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

0
votes
0answers
13 views

Big Data Use Cases

I am going to study about big data use cases in the real world. I want the stuff to be preferable in book format(not article or blog posts or website articles) and high level, industry based and ...
-1
votes
2answers
34 views

How do you draw a random sample of unique IDs in a large dataset?

Rookie here -- I have a a large data set with about 75,000 observations, and 2000 unique IDs. Therefore, each unique ID has about 37 observations. I'm trying to draw a random sample of unique IDs, say ...
0
votes
0answers
21 views

Comparing the distribution (or properties of distributions) of large datasets for practically significant differences

I have several populations with a large n (100,000 - 1M+ samples), which i wish to compare. I know that virtually any statistical test i apply will reject the null hypothesis due to the extreme power ...
1
vote
0answers
57 views

Calculation of an overall top X over several groups

I'm not a statistics expert, so I'm hoping someone here can lend me a hand. I've got a bunch of key-value pairs associated with a specific time range. Something like this: ...
1
vote
1answer
85 views

Comparing large categorical data sets with low or zero counts

I'm dealing with a biological feature which can be classified into $2^{20}$ categories. I also have two pretty large data sets of 1 and 3 million entries. Actually, only around 30 thousand categories ...
4
votes
2answers
102 views

How to reduce the dimension of $10^8$ vectors

I have $10^8$ vectors in $1000$ dimensions each. I would like to drastically reduce their dimensions. However PCA seems computationally infeasible. Are there near linear time methods to do ...
1
vote
1answer
56 views

Best metric for evaluation of mixture-of-Gaussian clusters on big-data

I have made a new algorithm that is specifically crafted for clustering very large datasets. In order to document it as a research paper, I have to choose one or two internal (no-label) cluster ...
1
vote
0answers
13 views

Sybil detection metric

I have a set of user data and I want to build some kind of metric to evaluate the probability of the user being a sybil (a "fake" account). But I have a very limited set of users who are sybils with ...
2
votes
1answer
21 views

Need reasons or references on small p-values with large data sets

What are reasons or references for a statement such as "When you have a lot of data the statistical problem you run into is that even a tiny difference will be statistically significant." I see this ...
0
votes
1answer
41 views

Random forest ML algorithm suitable for use on cluster based HPC?

I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve, the model contains over 12gb of data, including over 500 parameters. Now running small ...
1
vote
1answer
37 views

Testing for heteroscedasticity with many observations

I used the Breusch-Pagan test for heteroscedasticity, but I have many observations ($\approx 500,\! 000$) and the Breusch-Pagan test uses $nR^2$ as a test statistic where $n$ is the number of ...
0
votes
1answer
22 views

Literature for Cross Validation on Sparse Data?

I've read a lot about Cross Validation to estimate prediction error, specifically for selecting the number of components in a PCA model (I'm not doing SVD/PCA, but it's very similar), but I can't find ...
1
vote
0answers
48 views

regression algorithms which work with sparse categorical predictors

I am working with a very sparse problem with a large number of categories per feature and I am currently looking for existing machine learning regression algorithm implementations which can either ...
0
votes
1answer
32 views

Memory consumption in NLP tasks [closed]

I am trying to apply a simple Naive Bayes or SVM (libSVM) algorithms to a large data set, which I've constructed as an .arff file. The number of features in my set is ~180k and there are ~6k ...
0
votes
1answer
72 views

Can I subsample a large dataset at every MCMC iteration?

I have a large dataset from which I want to perform a bayesian probit regression using Gibbs sampling 1. Since the dataset has one milion rows, and variables from a truncated normal must be sampled ...
1
vote
0answers
24 views

Window models in stream data processing

Reading about data stream clustering I met the next terms: landmark window model, sliding window model, damped window. As to sliding window it's clear - oldest data escape the scope, the new data ...
1
vote
1answer
56 views

Comparing two distributions

I have more than five hundred thousands samples of a continuous variable measured in two groups: a treatment and a control one. I would like to decide whether the measurements follow the same ...
-1
votes
1answer
82 views

Clustering large movie dataset using k-medoids?

I have to cluster a movie dataset of 10000 movies. A movie has attributes like Genres, Actors, Directors, Year. Earlier I thought that we can use a simple clustering algorithm like k-medoids and the ...
2
votes
0answers
90 views

Is Big Data the end of Statistics? [closed]

Statistics emerged as a discipline whose goal was to allow inferences about inaccessible populations from data obtained from its samples. Does modern access to huge amounts of data depreciate the ...
3
votes
1answer
49 views

Dealing with Big Data and Lots of Variables

What is a good technique to use on data that has many categorical variables with many possible values? For example, let's say you are trying to determine what kind of people are more likely to ...
0
votes
2answers
111 views

Streaming k-means

I want to perform something like streaming/online/out-of-core kmeans clustering on large data. Here is simple idea: Break all data into N chunks. Read from disk 1st chunk and calculate centroids ...
2
votes
0answers
68 views

Big Data? Have we solved Small Data yet? [closed]

There has been a lot of attention on Big Data recently, where the problems are often more logistical (how to deal with large volumes of data) rather than statistical. At the other end of the spectrum ...
5
votes
3answers
796 views

Is visual inspection the only way to compare large datasets?

I have two large data sets, in fact, one of them is even much larger than the other. Visually, there doesn't seem to be that much difference between them: The actual data underlying the box plot ...
1
vote
1answer
59 views

Alternative to spherical K-Means for clustering large high dimensional dataset

What are some alternatives to Spherical K-Means for clustering very large datasets of high dimension? I'm looking for something that will be fast even on large datasets, and preferably will not ...
4
votes
1answer
303 views

Testing large dataset for normality - how and is it reliable?

I'm examining a part of my dataset containing 46840 double values ranging from 1 to 1690 grouped in two groups. In order to analyze the differences between these groups I started by examining the ...
2
votes
0answers
57 views

Kolmogorov-Smirnov test for large sample [duplicate]

I am investigating if an exponential distribution is a good fit for a large sample of data (200) I have. I have already looked at a histogram but was wanting to investigate further. I was going to use ...
0
votes
0answers
25 views

Clustering algorithm for my situation?

Here is my situation. I have a corpus of over 500,000 news. Now I need to cluster the news based on closeness in time and cosine similarity, using vector-space model and TF-IDF weights. I want to ...
1
vote
1answer
95 views

Impact of data dimensionality on computation complexity of SVM?

What is the impact of data dimensionality on computation complexity of SVM? I found on the literature that the complexity of SVM is $O(N^3)$, where $N$ is the number of training examples. If the ...
0
votes
0answers
16 views

Are there any benchmarks comparing Mahout with older software?

I want to know about Mahout's implementation of classification algorithms - how does its accuracy compare with other large data software? Anyone knows a reference on this? I know about this SAS ...
1
vote
2answers
140 views

Predict only the first N principal components in a PCA analysis

I'm using R to analyze a very large dataset. I conduct a PCA on one dataset, PCA <- prcomp(formula = ~., data = train, scale = T, na.action=na.exclude) and ...
2
votes
2answers
293 views

How to summarize and understand the reults of DBSCAN clustering on big data?

Many clustering algorithms can be used with big data, eg. versions of KMeans, DBSCAN based on Hadoop, etc. But, with k means we will get k centroids for k clusters and we can map them to the space and ...
4
votes
2answers
422 views

What tools do Machine Learning experts use in the real world?

I'm currently taking a class covering some topics in machine learning. The class is taught in MATLAB using Liblinear so far. I was curious though what kind of tools people used in the real world to ...
0
votes
0answers
24 views

Large scale 1-Dimensional Gaussian Process Classification

I have a dataset with a small number of input points (e.g. +- 300), but millions of boolean outcomes. ...
8
votes
2answers
504 views

Why would a statistical model overfit if given a huge data set?

My current project may require me to build a model to predict the behavior of a certain group of people. the training data set contains only 6 variables (id is only for identification purposes): ...
0
votes
1answer
76 views

Sparse PCA/Dictionary learning when the features are extremely sparse?

I am trying to do sparse PCA/dictionary learning, that is decompose a matrix $X\approx UV$ where the loading matrix $V$ is sparse, usually enforced with an $\ell_1$ penalty (the difference between ...
4
votes
0answers
48 views

Splitting a variable with nominal and numeric values

I have a variable that has both numeric and nominal components. The source has a documentation which helps in identifying which is which and for splitting into their proper components. I will do ...
0
votes
3answers
86 views

Statistical comparisons between large data sets

Currently, I am looking for the correct (or suitable) statistical method to compare 4 very large datasets (n = 31 million each), that are based on an experiment where a continuous variable was ...
1
vote
0answers
78 views

What is a good and efficient algorithm for a content based recommender?

I want to build a content based recommender in a restricted environment regarding cpu power and memory (to be specific: a mobile device, but it is not acceptable to build the recommender on a remote ...
0
votes
1answer
42 views

Question regarding significance level [duplicate]

Say, for instance, I'm estimating a model with about 5 million observations using linear regression or MLE. Given that the estimates are consistent, using the standard rule of rejecting the null on a ...
1
vote
1answer
326 views

What data structure to use for my cluster analysis or what cluster analysis to use for my data?

I have a large dataset of categorical variables. The data consists of shoppers who purchased two items during a single trip to a store. There are approximately 75,000 cases and 1,500 different ...
6
votes
1answer
178 views

Star Coordinates vs. principal component analysis

I currently preparing a presentation for a university course in "Visual Data analysis". And one of my topics is the "Star Coordinate" visualization. Star Coordinates As Star Coordinates perform a ...
1
vote
1answer
122 views

Recursive logistic regression merge

I need to make regression on big amount of data, each row have around 1000 features. Did will outcome will be same or better when i make 4 separate regressions of 250 features and after that i will ...
4
votes
4answers
262 views

How do I deal with large data similarity computation?

I have lot of records like this: M is about 10 million and N is about 100K. Now I want to apply collaborative filtering on ...
1
vote
3answers
108 views

Large data analysis - learning resources

My question is very simple: which learning resources (books, courses, online courses, and so on) about "large data analysis" would you suggest to a graduate with a strong background in Machine ...
0
votes
2answers
270 views

Statistical Significance with large data sets [closed]

When I was a Ph.D. student I was trained in no uncertain terms that When we had large numbers of data results, that the number of significant results HAD to be, in and of themselves SIGNIFICANT! I ...
2
votes
1answer
306 views

Non-normality of residuals in linear regression of very large sample in SPSS

I have a dataset of ~17,000 cases in SPSS 21 with which I am trying to run multiple linear regression. I have plotted the Studentised residuals against the unstandardised predicted values and also ...
0
votes
1answer
194 views

Multinomial logistic regression for big data

How do I go about doing a multinomial logistic regression when I have 70 million observations? Is it feasible? It seems that R is out of the question due to memory constraints?
2
votes
1answer
1k views

Mann-Whitney U test with very large sample size?

I'm doing a Mann-Whitney U test to compare two very large samples (sample size 1 = 13250; samlple size 2 = 38871) originating from a raster image. I know t-tests are not recommended to compare ...
4
votes
2answers
414 views

Comparing nested binary logistic regression models when $n$ is large

To better ask my question, I have provided some of the outputs from both a 16 variable model (fit) and a 17 variable model (...
1
vote
2answers
181 views

ANOVA with huge dataset - use only the mean for each condition?

I need advice about how to carry out an ANOVA. I studied some theory of ANOVA, but apparently it is not enough. Basically I collected around 300 reaction times for 12 subjects in my experiment. For ...