'Large data' refers to situations where the number of observations is so large that it necessitates changes in the way the the data analyst thinks about or conducts the analysis.

learn more… | top users | synonyms (2)

2
votes
0answers
23 views

Dealing with Big Data and Lots of Variables

What is a good technique to use on data that has many categorical variables with many possible values? For example, let's say you are trying to determine what kind of people are more likely to ...
0
votes
2answers
33 views

Streaming k-means

I want to perform something like streaming/online/out-of-core kmeans clustering on large data. Here is simple idea: Break all data into N chunks. Read from disk 1st chunk and calculate centroids ...
2
votes
0answers
52 views

Big Data? Have we solved Small Data yet? [closed]

There has been a lot of attention on Big Data recently, where the problems are often more logistical (how to deal with large volumes of data) rather than statistical. At the other end of the spectrum ...
5
votes
3answers
743 views

Is visual inspection the only way to compare large datasets?

I have two large data sets, in fact, one of them is even much larger than the other. Visually, there doesn't seem to be that much difference between them: The actual data underlying the box plot ...
1
vote
1answer
34 views

Alternative to spherical K-Means for clustering large high dimensional dataset

What are some alternatives to Spherical K-Means for clustering very large datasets of high dimension? I'm looking for something that will be fast even on large datasets, and preferably will not ...
4
votes
1answer
239 views

Testing large dataset for normality - how and is it reliable?

I'm examining a part of my dataset containing 46840 double values ranging from 1 to 1690 grouped in two groups. In order to analyze the differences between these groups I started by examining the ...
2
votes
0answers
55 views

Kolmogorov-Smirnov test for large sample [duplicate]

I am investigating if an exponential distribution is a good fit for a large sample of data (200) I have. I have already looked at a histogram but was wanting to investigate further. I was going to use ...
0
votes
0answers
22 views

Clustering algorithm for my situation?

Here is my situation. I have a corpus of over 500,000 news. Now I need to cluster the news based on closeness in time and cosine similarity, using vector-space model and TF-IDF weights. I want to ...
1
vote
1answer
46 views

Impact of data dimensionality on computation complexity of SVM?

What is the impact of data dimensionality on computation complexity of SVM? I found on the literature that the complexity of SVM is $O(N^3)$, where $N$ is the number of training examples. If the ...
0
votes
0answers
11 views

Are there any benchmarks comparing Mahout with older software?

I want to know about Mahout's implementation of classification algorithms - how does its accuracy compare with other large data software? Anyone knows a reference on this? I know about this SAS ...
1
vote
2answers
64 views

Predict only the first N principal components in a PCA analysis

I'm using R to analyze a very large dataset. I conduct a PCA on one dataset, PCA <- prcomp(formula = ~., data = train, scale = T, na.action=na.exclude) and ...
2
votes
2answers
104 views

How to summarize and understand the reults of DBSCAN clustering on big data?

Many clustering algorithms can be used with big data, eg. versions of KMeans, DBSCAN based on Hadoop, etc. But, with k means we will get k centroids for k clusters and we can map them to the space and ...
4
votes
2answers
274 views

What tools do Machine Learning experts use in the real world?

I'm currently taking a class covering some topics in machine learning. The class is taught in MATLAB using Liblinear so far. I was curious though what kind of tools people used in the real world to ...
0
votes
0answers
21 views

Large scale 1-Dimensional Gaussian Process Classification

I have a dataset with a small number of input points (e.g. +- 300), but millions of boolean outcomes. ...
8
votes
2answers
428 views

Why would a statistical model overfit if given a huge data set?

My current project may require me to build a model to predict the behavior of a certain group of people. the training data set contains only 6 variables (id is only for identification purposes): ...
0
votes
1answer
32 views

Sparse PCA/Dictionary learning when the features are extremely sparse?

I am trying to do sparse PCA/dictionary learning, that is decompose a matrix $X\approx UV$ where the loading matrix $V$ is sparse, usually enforced with an $\ell_1$ penalty (the difference between ...
4
votes
0answers
41 views

Splitting a variable with nominal and numeric values

I have a variable that has both numeric and nominal components. The source has a documentation which helps in identifying which is which and for splitting into their proper components. I will do ...
0
votes
3answers
65 views

Statistical comparisons between large data sets

Currently, I am looking for the correct (or suitable) statistical method to compare 4 very large datasets (n = 31 million each), that are based on an experiment where a continuous variable was ...
1
vote
0answers
76 views

What is a good and efficient algorithm for a content based recommender?

I want to build a content based recommender in a restricted environment regarding cpu power and memory (to be specific: a mobile device, but it is not acceptable to build the recommender on a remote ...
0
votes
1answer
32 views

Question regarding significance level [duplicate]

Say, for instance, I'm estimating a model with about 5 million observations using linear regression or MLE. Given that the estimates are consistent, using the standard rule of rejecting the null on a ...
1
vote
1answer
113 views

What data structure to use for my cluster analysis or what cluster analysis to use for my data?

I have a large dataset of categorical variables. The data consists of shoppers who purchased two items during a single trip to a store. There are approximately 75,000 cases and 1,500 different ...
6
votes
1answer
144 views

Star Coordinates vs. principal component analysis

I currently preparing a presentation for a university course in "Visual Data analysis". And one of my topics is the "Star Coordinate" visualization. Star Coordinates As Star Coordinates perform a ...
1
vote
1answer
79 views

Recursive logistic regression merge

I need to make regression on big amount of data, each row have around 1000 features. Did will outcome will be same or better when i make 4 separate regressions of 250 features and after that i will ...
3
votes
4answers
132 views

How do I deal with large data similarity computation?

I have lot of records like this: M is about 10 million and N is about 100K. Now I want to apply collaborative filtering on ...
1
vote
3answers
91 views

Large data analysis - learning resources

My question is very simple: which learning resources (books, courses, online courses, and so on) about "large data analysis" would you suggest to a graduate with a strong background in Machine ...
0
votes
2answers
153 views

Statistical Significance with large data sets [closed]

When I was a Ph.D. student I was trained in no uncertain terms that When we had large numbers of data results, that the number of significant results HAD to be, in and of themselves SIGNIFICANT! I ...
2
votes
1answer
157 views

Non-normality of residuals in linear regression of very large sample in SPSS

I have a dataset of ~17,000 cases in SPSS 21 with which I am trying to run multiple linear regression. I have plotted the Studentised residuals against the unstandardised predicted values and also ...
0
votes
1answer
105 views

Multinomial logistic regression for big data

How do I go about doing a multinomial logistic regression when I have 70 million observations? Is it feasible? It seems that R is out of the question due to memory constraints?
1
vote
1answer
635 views

Mann-Whitney U test with very large sample size?

I'm doing a Mann-Whitney U test to compare two very large samples (sample size 1 = 13250; samlple size 2 = 38871) originating from a raster image. I know t-tests are not recommended to compare ...
4
votes
2answers
303 views

Comparing nested binary logistic regression models when $n$ is large

To better ask my question, I have provided some of the outputs from both a 16 variable model (fit) and a 17 variable model (...
1
vote
2answers
135 views

ANOVA with huge dataset - use only the mean for each condition?

I need advice about how to carry out an ANOVA. I studied some theory of ANOVA, but apparently it is not enough. Basically I collected around 300 reaction times for 12 subjects in my experiment. For ...
1
vote
1answer
73 views

What is a high dimensional multivariate data set?

I'm new to data analysis and data mining. Often in the papers I'm reading, they use the term "high dimensional multivariate data set." Currently, my task is to detect an outlier and visualize the same ...
0
votes
1answer
52 views

Multidimensional scaling for big dissimilarity matrix

I have a large symmetrical dissimilarity matrix of dimension 300 000. Can you please suggest the multidimensional scaling algorithms that can work with such large data? Input of course can be the ...
2
votes
1answer
154 views

What is the time complexity of Lasso regression

What is the asymptotic time complexity of Lasso regression as either the number of rows or columns grows?
9
votes
4answers
152 views

Goodness-of-fit for very large sample sizes

I collect very large samples (>1,000,000) of categorical data each day and want to see the data looks "significantly" different between days to detect errors in data collection. I thought using a ...
3
votes
2answers
109 views

Estimating ROC/AUC on large data sets?

Plotting an ROC curve of a classifier compared to cases requires that the data set be sorted first on the classifier score. I am in a position where I need to calculate ROC on a large data set very ...
3
votes
1answer
102 views

Large scale ridge regression

I'm trying to solve a problem of the form $\min_x \frac{1}{2}||Ax-b||^2_2 + \frac{\rho}{2}||x-z||^2_F$ where both $x$ and $b$ are high dimensional, and $b$ is much higher dimensional than $x$. The ...
0
votes
0answers
30 views

Can I use the variance of a set of observation as heuristic to decide how many times repeat an experiment?

I am applying a clustering algorithm (K-means) to a huge set of high dimensional data points (SIFT descriptors). The algorithm is not deterministic and its results depend on the initialization of the ...
1
vote
0answers
97 views

Netflix Challenge - some help with SVD/SoftImpute

I'm currently working on the Netflix Challenge with the original huge dataset and have run into some problems. I don't have access to any servers or computing clusters so I've been running everything ...
0
votes
0answers
33 views

Reducing data size to cross correlate with another data set?

I have three matrices A, B, C. A is a matrix of 200 X 32. There are 2000 such different A matrices which make up the B matrix. B is a matrix of 2000 x A. That is there are 2000 x 200 rows in matrix B ...
5
votes
0answers
88 views

Similarity algorithms

Let's say I have 300 restaurants that I want to compare to each other on the basis of a "similarity score". To try and determine similarity scores, I pick a reference restaurant and pick 3 other ...
2
votes
2answers
155 views

Data cleaning for large sample data set in multiple linear regression

I have 70,000 observations for my depentant variable. I have 12 independant variables. After removing zero value and error and missing value form my data set, my data reduced to 4000. Can I still do ...
2
votes
1answer
102 views

Cluster many thousands observations (mixed variable types). Cluster subsample and then classify the rest observations?

I'm trying to run a cluster analysis on a large dataset (70k+ observations to cluster) with mixed variables (numeric, ordinal, binary and nominal). I don't think I can create the distance matrix using ...
2
votes
2answers
202 views

Machine learning on big data: capability of generalization

I have being working applying different ML Algorithms oriented to Big Data and I have some open questions that I find interesting to think about. One of the first lectures about statistics begins ...
3
votes
4answers
237 views

Which regression tree to use for large data?

I have a dataframe with 2 million rows and approximately 200 columns / features. Approximately 30-40% of the entries are blank. I am trying to find important features for a binary response variable. ...
6
votes
3answers
574 views

Which machine learning algorithms can be scaled using hadoop/map-reduce

Scalable machine learning algorithms seem like the buzz these days. Every company is handling nothing short of big data. Is there a textbook which discusses what machine learning algorithms can be ...
0
votes
1answer
95 views

How to deal with giant sparse matrices?

I'm looking to do some heavy-duty manipulation of some really large and often very sparse matrices and I'm looking for the right tool for the job. These matrices will be much, much larger than the RAM ...
3
votes
2answers
208 views

Does it make sense to compute confidence intervals and to test hypotheses when data from whole population is available?

Does it make sense to compute confidence intervals and to test hypotheses when data from whole population is available? In my opinion, the answer is no, since we can accurately compute true values of ...
1
vote
1answer
169 views

Correlation or independence on contingency table for large N

I have a dataset with about 35,000 individuals described by around 15 categorical variables. I'm trying to study the independence / correlation between these 15 categorical variables. My first idea ...
2
votes
3answers
245 views

Alternatives to stepwise logistic regression with LARGE datasets

After reviewing related questions on Cross Validated and countless articles and discussions regarding the inappropriate use of stepwise regression for variable selection, I am still unable to find the ...