'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

0
votes
0answers
17 views

Reading in data set [on hold]

I am trying to read in this data set using R http://statweb.stanford.edu/~tibs/ElemStatLearn/datasets/nci.data I've tried several different things. Can anyone help me out?
0
votes
0answers
16 views

appropriate sample size to fit non-normal distribution parameters

I have a population that, in theory, I could determine exactly. I am not doing so, however, because on my poor home computer, it would be too computationally intensive. Each calculation takes 45-90 ...
1
vote
1answer
26 views

Big Data Regression Coefficient Estimation

I am working on a very large data set (n = 6.5 million) and I am trying to come up with a simple linear regression between two variables. I am working in R and using a monte carlo style simulation to ...
2
votes
1answer
59 views

large y, small n

"Large p, small n" typically refers to "many independent variables, few samples". In my case, I have 1 independent variable, 300 dependent variables, and < 20 samples. Thus, my case is not the ...
1
vote
1answer
40 views

Can binning a continuous predictor or DV variable improve large data sets fit?

I read that averaging and binning a continuous predictor variable is in general a bad idea because it's always better to fit the continuous relationship through splines, poly and all of that. Sure, I ...
1
vote
1answer
34 views

Is it possible to fit a logistic regression model to a dataset with categorical predictive variables with very high number of levels each?

I want to fit a model to a very large dataset, with a standard binary response variable and with 3 categorical predictor variables with 3000, 15 and 2 levels. Is there any inherent problem in this ...
0
votes
0answers
9 views

Linear-time approximation to kernel SVM?

Scaling kernel support vector machines to large datasets is a very challenging problem. For linear SVMs, PEGASOS is able to learn efficiently online, so training time scales linearly with the size of ...
6
votes
1answer
122 views

What are some interesting examples of wrong or crazy inferences being drawn from Big Data?

I'm interested in well-known examples of Big Data misinterpreted, poorly analysed, or wrongly employed to unscientific and incorrect ends. Would appreciate any examples or observations. Thanks a lot!
2
votes
2answers
58 views

Does the dataset size influence a machine learning algorithm?

So, imagine having access to sufficient data (millions of datapoints for training and testing) of sufficient quality. Please ignore concept drift for now and assume the data static and does not change ...
3
votes
0answers
50 views

Permutation for large sample sizes (and z-test)

I was trying to do a permutation test on a large amount of temperature observations in R. Approximately 1700 temperature observations (g1) from one place and 2 ...
0
votes
0answers
18 views

Big Data Use Cases

I am going to study about big data use cases in the real world. I want the stuff to be preferable in book format(not article or blog posts or website articles) and high level, industry based and ...
-1
votes
2answers
44 views

How do you draw a random sample of unique IDs in a large dataset?

Rookie here -- I have a a large data set with about 75,000 observations, and 2000 unique IDs. Therefore, each unique ID has about 37 observations. I'm trying to draw a random sample of unique IDs, say ...
1
vote
0answers
58 views

Calculation of an overall top X over several groups

I'm not a statistics expert, so I'm hoping someone here can lend me a hand. I've got a bunch of key-value pairs associated with a specific time range. Something like this: ...
1
vote
1answer
99 views

Comparing large categorical data sets with low or zero counts

I'm dealing with a biological feature which can be classified into $2^{20}$ categories. I also have two pretty large data sets of 1 and 3 million entries. Actually, only around 30 thousand categories ...
4
votes
2answers
103 views

How to reduce the dimension of $10^8$ vectors

I have $10^8$ vectors in $1000$ dimensions each. I would like to drastically reduce their dimensions. However PCA seems computationally infeasible. Are there near linear time methods to do ...
1
vote
1answer
65 views

Best metric for evaluation of mixture-of-Gaussian clusters on big-data

I have made a new algorithm that is specifically crafted for clustering very large datasets. In order to document it as a research paper, I have to choose one or two internal (no-label) cluster ...
1
vote
0answers
13 views

Sybil detection metric

I have a set of user data and I want to build some kind of metric to evaluate the probability of the user being a sybil (a "fake" account). But I have a very limited set of users who are sybils with ...
2
votes
1answer
23 views

Need reasons or references on small p-values with large data sets

What are reasons or references for a statement such as "When you have a lot of data the statistical problem you run into is that even a tiny difference will be statistically significant." I see this ...
0
votes
1answer
61 views

Random forest ML algorithm suitable for use on cluster based HPC?

I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve, the model contains over 12gb of data, including over 500 parameters. Now running small ...
1
vote
1answer
48 views

Testing for heteroscedasticity with many observations

I used the Breusch-Pagan test for heteroscedasticity, but I have many observations ($\approx 500,\! 000$) and the Breusch-Pagan test uses $nR^2$ as a test statistic where $n$ is the number of ...
0
votes
1answer
28 views

Literature for Cross Validation on Sparse Data?

I've read a lot about Cross Validation to estimate prediction error, specifically for selecting the number of components in a PCA model (I'm not doing SVD/PCA, but it's very similar), but I can't find ...
1
vote
0answers
61 views

regression algorithms which work with sparse categorical predictors

I am working with a very sparse problem with a large number of categories per feature and I am currently looking for existing machine learning regression algorithm implementations which can either ...
0
votes
1answer
37 views

Memory consumption in NLP tasks [closed]

I am trying to apply a simple Naive Bayes or SVM (libSVM) algorithms to a large data set, which I've constructed as an .arff file. The number of features in my set is ~180k and there are ~6k ...
0
votes
1answer
91 views

Can I subsample a large dataset at every MCMC iteration?

I have a large dataset from which I want to perform a bayesian probit regression using Gibbs sampling 1. Since the dataset has one milion rows, and variables from a truncated normal must be sampled ...
1
vote
0answers
26 views

Window models in stream data processing

Reading about data stream clustering I met the next terms: landmark window model, sliding window model, damped window. As to sliding window it's clear - oldest data escape the scope, the new data ...
2
votes
2answers
133 views

Fast missing data imputation in R for big data that is more sophisticated than simply imputing the means?

I need a package for missing data imputation in R. But since I am dealing with big data, the number of missing data entries can also be high. The packages which impute using mean or median are of ...
1
vote
1answer
72 views

Comparing two distributions

I have more than five hundred thousands samples of a continuous variable measured in two groups: a treatment and a control one. I would like to decide whether the measurements follow the same ...
-1
votes
1answer
106 views

Clustering large movie dataset using k-medoids?

I have to cluster a movie dataset of 10000 movies. A movie has attributes like Genres, Actors, Directors, Year. Earlier I thought that we can use a simple clustering algorithm like k-medoids and the ...
2
votes
0answers
98 views

Is Big Data the end of Statistics? [closed]

Statistics emerged as a discipline whose goal was to allow inferences about inaccessible populations from data obtained from its samples. Does modern access to huge amounts of data depreciate the ...
3
votes
1answer
68 views

Dealing with Big Data and Lots of Variables

What is a good technique to use on data that has many categorical variables with many possible values? For example, let's say you are trying to determine what kind of people are more likely to ...
0
votes
2answers
133 views

Streaming k-means

I want to perform something like streaming/online/out-of-core kmeans clustering on large data. Here is simple idea: Break all data into N chunks. Read from disk 1st chunk and calculate centroids ...
3
votes
0answers
75 views

Big Data? Have we solved Small Data yet? [closed]

There has been a lot of attention on Big Data recently, where the problems are often more logistical (how to deal with large volumes of data) rather than statistical. At the other end of the spectrum ...
5
votes
3answers
816 views

Is visual inspection the only way to compare large datasets?

I have two large data sets, in fact, one of them is even much larger than the other. Visually, there doesn't seem to be that much difference between them: The actual data underlying the box plot ...
1
vote
1answer
64 views

Alternative to spherical K-Means for clustering large high dimensional dataset

What are some alternatives to Spherical K-Means for clustering very large datasets of high dimension? I'm looking for something that will be fast even on large datasets, and preferably will not ...
4
votes
1answer
328 views

Testing large dataset for normality - how and is it reliable?

I'm examining a part of my dataset containing 46840 double values ranging from 1 to 1690 grouped in two groups. In order to analyze the differences between these groups I started by examining the ...
2
votes
0answers
57 views

Kolmogorov-Smirnov test for large sample [duplicate]

I am investigating if an exponential distribution is a good fit for a large sample of data (200) I have. I have already looked at a histogram but was wanting to investigate further. I was going to use ...
0
votes
0answers
26 views

Clustering algorithm for my situation?

Here is my situation. I have a corpus of over 500,000 news. Now I need to cluster the news based on closeness in time and cosine similarity, using vector-space model and TF-IDF weights. I want to ...
1
vote
1answer
104 views

Impact of data dimensionality on computation complexity of SVM?

What is the impact of data dimensionality on computation complexity of SVM? I found on the literature that the complexity of SVM is $O(N^3)$, where $N$ is the number of training examples. If the ...
0
votes
0answers
16 views

Are there any benchmarks comparing Mahout with older software?

I want to know about Mahout's implementation of classification algorithms - how does its accuracy compare with other large data software? Anyone knows a reference on this? I know about this SAS ...
1
vote
2answers
185 views

Predict only the first N principal components in a PCA analysis

I'm using R to analyze a very large dataset. I conduct a PCA on one dataset, PCA <- prcomp(formula = ~., data = train, scale = T, na.action=na.exclude) and ...
2
votes
2answers
372 views

How to summarize and understand the reults of DBSCAN clustering on big data?

Many clustering algorithms can be used with big data, eg. versions of KMeans, DBSCAN based on Hadoop, etc. But, with k means we will get k centroids for k clusters and we can map them to the space and ...
4
votes
2answers
494 views

What tools do Machine Learning experts use in the real world?

I'm currently taking a class covering some topics in machine learning. The class is taught in MATLAB using Liblinear so far. I was curious though what kind of tools people used in the real world to ...
0
votes
0answers
25 views

Large scale 1-Dimensional Gaussian Process Classification

I have a dataset with a small number of input points (e.g. +- 300), but millions of boolean outcomes. ...
8
votes
2answers
535 views

Why would a statistical model overfit if given a huge data set?

My current project may require me to build a model to predict the behavior of a certain group of people. the training data set contains only 6 variables (id is only for identification purposes): ...
0
votes
1answer
88 views

Sparse PCA/Dictionary learning when the features are extremely sparse?

I am trying to do sparse PCA/dictionary learning, that is decompose a matrix $X\approx UV$ where the loading matrix $V$ is sparse, usually enforced with an $\ell_1$ penalty (the difference between ...
4
votes
0answers
49 views

Splitting a variable with nominal and numeric values

I have a variable that has both numeric and nominal components. The source has a documentation which helps in identifying which is which and for splitting into their proper components. I will do ...
0
votes
3answers
93 views

Statistical comparisons between large data sets

Currently, I am looking for the correct (or suitable) statistical method to compare 4 very large datasets (n = 31 million each), that are based on an experiment where a continuous variable was ...
1
vote
0answers
78 views

What is a good and efficient algorithm for a content based recommender?

I want to build a content based recommender in a restricted environment regarding cpu power and memory (to be specific: a mobile device, but it is not acceptable to build the recommender on a remote ...
0
votes
1answer
46 views

Question regarding significance level [duplicate]

Say, for instance, I'm estimating a model with about 5 million observations using linear regression or MLE. Given that the estimates are consistent, using the standard rule of rejecting the null on a ...
1
vote
1answer
418 views

What data structure to use for my cluster analysis or what cluster analysis to use for my data?

I have a large dataset of categorical variables. The data consists of shoppers who purchased two items during a single trip to a store. There are approximately 75,000 cases and 1,500 different ...