Tagged Questions

'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

0
votes
0answers
13 views

Create a density map with latitude and longitude

I am working on a data mining project for my class and would like to visualize some data. I basically have 20,000 instances and would like to create a density map based on house value. I have ...
1
vote
0answers
8 views

Joint significance testing w/ large samples

long time listener, first time caller. Unfortunately I can't show code as the computer the analysis is done on has rather tight security. I need to test for joint significance in a logit model that ...
3
votes
1answer
58 views

Robust estimates of the covariance matrix in the big data space

I am trying to compute the robust estimates of the covariance matrix (and also the mean) in the big data space. I am aware of FastMVE and FastMCD (Minimum Covariance Determinant and Minimum Volume ...
2
votes
0answers
52 views

How to fit OLS with many categorical levels, on more than one category

This question is not meant to be a software question, but I will illustrate the issue using R a bit. My Understanding of the Simple Case If I have a simple linear model with a categorical variable ...
0
votes
0answers
40 views

Quantifying the predictive ability of a model developed from a huge data set? (variation of bootstrapping?)

I have a statistical model with around 20 predictor variables, built on 90% of a dataset consisting of over 600k observations. The original developer held out 10% of the original dataset for the ...
1
vote
0answers
42 views

Wherefore Big Data?

It's been many years since I've done any statistics (or any serious math), but I do remember that the sampling error decreases more slowly for larger sample sizes (like n^-1/2, at least for some ...
0
votes
0answers
13 views

Difficulties of making inferences on large and high dimensional data

I've seen a couple of seemingly unrelated notes about working with large volumes of data and it struck me that I couldn't find much content on problems specific to statistical analysis of Big Data. ...
0
votes
0answers
44 views

Identifying multivariate outliers in a large sample with missing data, using SPSS

I'm a psychology PhD student doing analysis on a relatively large set of data, obtained via online surveys. The purpose of the study is largely to determine normative data for a population of adults, ...
0
votes
0answers
22 views

appropriate sample size to fit non-normal distribution parameters

I have a population that, in theory, I could determine exactly. I am not doing so, however, because on my poor home computer, it would be too computationally intensive. Each calculation takes 45-90 ...
1
vote
1answer
28 views

Big Data Regression Coefficient Estimation

I am working on a very large data set (n = 6.5 million) and I am trying to come up with a simple linear regression between two variables. I am working in R and using a monte carlo style simulation to ...
1
vote
1answer
49 views

Can binning a continuous predictor or DV variable improve large data sets fit?

I read that averaging and binning a continuous predictor variable is in general a bad idea because it's always better to fit the continuous relationship through splines, poly and all of that. Sure, I ...
1
vote
1answer
39 views

Is it possible to fit a logistic regression model to a dataset with categorical predictive variables with very high number of levels each?

I want to fit a model to a very large dataset, with a standard binary response variable and with 3 categorical predictor variables with 3000, 15 and 2 levels. Is there any inherent problem in this ...
0
votes
0answers
10 views

Linear-time approximation to kernel SVM?

Scaling kernel support vector machines to large datasets is a very challenging problem. For linear SVMs, PEGASOS is able to learn efficiently online, so training time scales linearly with the size of ...
6
votes
1answer
138 views

What are some interesting examples of wrong or crazy inferences being drawn from Big Data?

I'm interested in well-known examples of Big Data misinterpreted, poorly analysed, or wrongly employed to unscientific and incorrect ends. Would appreciate any examples or observations. Thanks a lot!
2
votes
2answers
77 views

Does the dataset size influence a machine learning algorithm?

So, imagine having access to sufficient data (millions of datapoints for training and testing) of sufficient quality. Please ignore concept drift for now and assume the data static and does not change ...
3
votes
0answers
60 views

Permutation for large sample sizes (and z-test)

I was trying to do a permutation test on a large amount of temperature observations in R. Approximately 1700 temperature observations (g1) from one place and 2 ...
0
votes
0answers
18 views

Big Data Use Cases

I am going to study about big data use cases in the real world. I want the stuff to be preferable in book format(not article or blog posts or website articles) and high level, industry based and ...
-1
votes
2answers
51 views

How do you draw a random sample of unique IDs in a large dataset?

Rookie here -- I have a a large data set with about 75,000 observations, and 2000 unique IDs. Therefore, each unique ID has about 37 observations. I'm trying to draw a random sample of unique IDs, say ...
1
vote
0answers
58 views

Calculation of an overall top X over several groups

I'm not a statistics expert, so I'm hoping someone here can lend me a hand. I've got a bunch of key-value pairs associated with a specific time range. Something like this: ...
1
vote
1answer
104 views

Comparing large categorical data sets with low or zero counts

I'm dealing with a biological feature which can be classified into $2^{20}$ categories. I also have two pretty large data sets of 1 and 3 million entries. Actually, only around 30 thousand categories ...
4
votes
2answers
103 views

How to reduce the dimension of $10^8$ vectors

I have $10^8$ vectors in $1000$ dimensions each. I would like to drastically reduce their dimensions. However PCA seems computationally infeasible. Are there near linear time methods to do ...
1
vote
1answer
73 views

Best metric for evaluation of mixture-of-Gaussian clusters on big-data

I have made a new algorithm that is specifically crafted for clustering very large datasets. In order to document it as a research paper, I have to choose one or two internal (no-label) cluster ...
1
vote
0answers
13 views

Sybil detection metric

I have a set of user data and I want to build some kind of metric to evaluate the probability of the user being a sybil (a "fake" account). But I have a very limited set of users who are sybils with ...
2
votes
1answer
23 views

Need reasons or references on small p-values with large data sets

What are reasons or references for a statement such as "When you have a lot of data the statistical problem you run into is that even a tiny difference will be statistically significant." I see this ...
0
votes
1answer
66 views

Random forest ML algorithm suitable for use on cluster based HPC?

I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve, the model contains over 12gb of data, including over 500 parameters. Now running small ...
1
vote
1answer
50 views

Testing for heteroscedasticity with many observations

I used the Breusch-Pagan test for heteroscedasticity, but I have many observations ($\approx 500,\! 000$) and the Breusch-Pagan test uses $nR^2$ as a test statistic where $n$ is the number of ...
0
votes
1answer
31 views

Literature for Cross Validation on Sparse Data?

I've read a lot about Cross Validation to estimate prediction error, specifically for selecting the number of components in a PCA model (I'm not doing SVD/PCA, but it's very similar), but I can't find ...
1
vote
0answers
74 views

regression algorithms which work with sparse categorical predictors

I am working with a very sparse problem with a large number of categories per feature and I am currently looking for existing machine learning regression algorithm implementations which can either ...
0
votes
1answer
38 views

Memory consumption in NLP tasks [closed]

I am trying to apply a simple Naive Bayes or SVM (libSVM) algorithms to a large data set, which I've constructed as an .arff file. The number of features in my set is ~180k and there are ~6k ...
0
votes
1answer
108 views

Can I subsample a large dataset at every MCMC iteration?

I have a large dataset from which I want to perform a bayesian probit regression using Gibbs sampling 1. Since the dataset has one milion rows, and variables from a truncated normal must be sampled ...
1
vote
0answers
29 views

Window models in stream data processing

Reading about data stream clustering I met the next terms: landmark window model, sliding window model, damped window. As to sliding window it's clear - oldest data escape the scope, the new data ...
2
votes
2answers
151 views

Fast missing data imputation in R for big data that is more sophisticated than simply imputing the means?

I need a package for missing data imputation in R. But since I am dealing with big data, the number of missing data entries can also be high. The packages which impute using mean or median are of ...
1
vote
1answer
108 views

Comparing two distributions

I have more than five hundred thousands samples of a continuous variable measured in two groups: a treatment and a control one. I would like to decide whether the measurements follow the same ...
-1
votes
1answer
122 views

Clustering large movie dataset using k-medoids?

I have to cluster a movie dataset of 10000 movies. A movie has attributes like Genres, Actors, Directors, Year. Earlier I thought that we can use a simple clustering algorithm like k-medoids and the ...
2
votes
0answers
102 views

Is Big Data the end of Statistics? [closed]

Statistics emerged as a discipline whose goal was to allow inferences about inaccessible populations from data obtained from its samples. Does modern access to huge amounts of data depreciate the ...
3
votes
1answer
80 views

Dealing with Big Data and Lots of Variables

What is a good technique to use on data that has many categorical variables with many possible values? For example, let's say you are trying to determine what kind of people are more likely to ...
0
votes
2answers
148 views

Streaming k-means

I want to perform something like streaming/online/out-of-core kmeans clustering on large data. Here is simple idea: Break all data into N chunks. Read from disk 1st chunk and calculate centroids ...
3
votes
0answers
75 views

Big Data? Have we solved Small Data yet? [closed]

There has been a lot of attention on Big Data recently, where the problems are often more logistical (how to deal with large volumes of data) rather than statistical. At the other end of the spectrum ...
5
votes
3answers
832 views

Is visual inspection the only way to compare large datasets?

I have two large data sets, in fact, one of them is even much larger than the other. Visually, there doesn't seem to be that much difference between them: The actual data underlying the box plot ...
1
vote
1answer
66 views

Alternative to spherical K-Means for clustering large high dimensional dataset

What are some alternatives to Spherical K-Means for clustering very large datasets of high dimension? I'm looking for something that will be fast even on large datasets, and preferably will not ...
4
votes
1answer
351 views

Testing large dataset for normality - how and is it reliable?

I'm examining a part of my dataset containing 46840 double values ranging from 1 to 1690 grouped in two groups. In order to analyze the differences between these groups I started by examining the ...
2
votes
0answers
57 views

Kolmogorov-Smirnov test for large sample [duplicate]

I am investigating if an exponential distribution is a good fit for a large sample of data (200) I have. I have already looked at a histogram but was wanting to investigate further. I was going to use ...
0
votes
0answers
26 views

Clustering algorithm for my situation?

Here is my situation. I have a corpus of over 500,000 news. Now I need to cluster the news based on closeness in time and cosine similarity, using vector-space model and TF-IDF weights. I want to ...
1
vote
1answer
112 views

Impact of data dimensionality on computation complexity of SVM?

What is the impact of data dimensionality on computation complexity of SVM? I found on the literature that the complexity of SVM is $O(N^3)$, where $N$ is the number of training examples. If the ...
0
votes
0answers
16 views

Are there any benchmarks comparing Mahout with older software?

I want to know about Mahout's implementation of classification algorithms - how does its accuracy compare with other large data software? Anyone knows a reference on this? I know about this SAS ...
1
vote
2answers
246 views

Predict only the first N principal components in a PCA analysis

I'm using R to analyze a very large dataset. I conduct a PCA on one dataset, PCA <- prcomp(formula = ~., data = train, scale = T, na.action=na.exclude) and ...
2
votes
2answers
429 views

How to summarize and understand the reults of DBSCAN clustering on big data?

Many clustering algorithms can be used with big data, eg. versions of KMeans, DBSCAN based on Hadoop, etc. But, with k means we will get k centroids for k clusters and we can map them to the space and ...
4
votes
2answers
557 views

What tools do Machine Learning experts use in the real world?

I'm currently taking a class covering some topics in machine learning. The class is taught in MATLAB using Liblinear so far. I was curious though what kind of tools people used in the real world to ...
0
votes
0answers
26 views

Large scale 1-Dimensional Gaussian Process Classification

I have a dataset with a small number of input points (e.g. +- 300), but millions of boolean outcomes. ...
8
votes
2answers
566 views

Why would a statistical model overfit if given a huge data set?

My current project may require me to build a model to predict the behavior of a certain group of people. the training data set contains only 6 variables (id is only for identification purposes): ...