'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

0
votes
0answers
4 views

Apache Mahout Classification [on hold]

i need a corpus to try mahout classification, i've tried the AG's corpus of news articles downloaded from this site http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html but that was not ...
0
votes
0answers
14 views

Unsure which tests to use now that my continuous symptom scale has been replaced with its underlying subscales

I've been working with a large data set in SPSS, mostly focused on the symptom scales of one developmental disorder. I have data from two separate time points, a few years apart. So for example I've ...
0
votes
0answers
31 views

Associating non-linear three-time-point change with a continuous variable

I would be incredibly grateful for help or advice regarding my following project: I have 3 time points (0, 30, 120 min) and complete data for about $n=500$ subjects for a continuous variable $M$. ...
12
votes
4answers
310 views

Confidence intervals when the sample size is very large

My question could be rephrased as "how to assess a sampling error using big data", especially for a journal publication. Here is an example to illustrate a challenge. From a very large dataset ...
0
votes
0answers
20 views

Variable importance in classification?

For example: I have 100 books with 1000 words each. They belong to different classes (comedy,drama,...). Each class consist of 15 different books. When i do TDIDF (term frequency - inverse document ...
0
votes
0answers
26 views

Checking assumptions of a large sample - extremely confused

I'm working with a sample size of over 2000 and I have become extremely confused at the first hurdle... I plan to run three linear regressions and one logistic regression. If I was working with a ...
1
vote
0answers
23 views

Instances of sparse covariance matrices

I am trying to find large datasets with inherently sparse covariance matrices, to be tested with our algorithm. Basically, we will take the sample covariance matrix and enforce some structured ...
0
votes
0answers
25 views

Large samples and the p-values almost zero in GEE models

I’m trying to find correlations between some environmental variables and a binary variable, berried/not berried females (rock lobster). I have 20 years of data that were sampled on a daily basis, ...
1
vote
1answer
27 views

Practical collaborative filtering application for large database

I’m designing an item-based collaborative filtering for a large database with over 100,000 items. My question is how the whole process works in practice since the algorithm takes a long time to ...
4
votes
1answer
39 views

Distributed Datasets and MLE

Suppose I have a very large dataset of size N, evenly distributed over M computers so that each computer has N/M data points. Suppose I want to fit a model using MLE that requires an iterative method. ...
2
votes
1answer
62 views

Big data database + software for advanced statistical analysis?

I need to run some statistical hypothesis testing, Anova, student's, least square fit, median, data mining, clustering... on a very large quantity of distributed data. (>100TB, Maybe columnar or ...
3
votes
1answer
60 views

Should I use Mean Square Error or Classification Rate?

I am a self-taught person and I would like your help. I am learning about predictive modeling in general, and I'm also trying to do predictive modeling for a specific problem. I am exploring ...
4
votes
2answers
107 views

Can I use likelihood-ratio test to compare two samples drawn from power-law distributions?

I have to compare two large samples ($N = 10^{6}$) of discrete data drawn from power-law distributions to assess whether they are significantly different. I can't do that by means of a two-sample ...
0
votes
0answers
19 views

sklearn - Multinomial Naive Bayes (data too big???!!!)

I wanted to really understand the rationale behind the following code as written in python sklearn's manual partial_fit(X, y, classes=None, sample_weight=None) when the data is too big to fit in ...
0
votes
0answers
17 views

MARS for big data

I have a question, is there a way in which MARS/EARTH can work with ff objects or big data? I say this because I need an additive model for a sample of size 20000 and around 100 variables. I can not ...
1
vote
0answers
33 views

Feature space reduction for tag prediction

[x-post] from stackoverflow. I am writing a ML module (python) to predict tags for a stackoverflow question (tag + body). My corpus is of around 5 million questions with title, body and tags for ...
0
votes
1answer
56 views

How to apply an R prediction model to very big data from SQL database in parallel.

I dont need to load the entire dataset into memory. In fact I only need 1 row at a time to apply a trained model, get the predicted response and put that response somewhere, possibly back into another ...
3
votes
1answer
49 views

Is an F-test for equality of variance appropriate for a very large dataset?

I have a dataset with about 500,000 subjects and I am trying to establish whether the variance is equal. I first performed an F-test but then I realised the data is slightly skewed with kurtosis. So ...
4
votes
3answers
154 views

Why do irrelevant regressors become statistically significant in large samples?

I am trying to better understand statistical significance, effect sizes and the like. I have a perception (perhaps its wrong) that even irrelevant regressors often become statistically significant in ...
3
votes
0answers
35 views

Propensity score matching with large data

I have a large healthcare claims database with 1.6 million subjects and I'm interested in doing a cohort study with propensity score matching. I have produced my propensity score with a logistic ...
0
votes
0answers
26 views

Enterprise Use of SAS & SPSS and Impact of Open Source Platforms (R & Python) [duplicate]

I'm curious to hear from people that use SAS & SPSS why a lot of the Fortune 100/500 enterprises still use SAS & SPSS despite the high cost of the license. Also would love to understand what ...
1
vote
1answer
86 views

Classification problem-Big Data and simple decision rules: logit regression, LDA, random forest, cond. trees, or something else?

This is a big data question from someone who is more accustomed to small data. I would like to develop some classification "rules of thumb," that is, some simple decision rules or a decision tree ...
3
votes
2answers
80 views

Collinearity in multivariate regression with huge amounts of data

Take the following example. I wish to predict physical performance as a function of height and weight. I already know weight negatively affects performance. Height also negatively affects performance, ...
0
votes
1answer
98 views

Large data variable selection

I'm looking for some methods of variable selection on large datasets.The number of variables are around 30-40, but the number of observations is quite large (around 36000000) Any methods which I ...
0
votes
0answers
32 views

Diference between glm and bigglm estimates

How does bigglm function in biglm package work for logistic regression? I thought that it is not possible to calculate LR on chunks of data and then merge results. Will glm and bigglm yield ...
0
votes
1answer
65 views

Compare two distributions of large sizes and unequal variances where one distribution is heavily skewed

My data is from cells that are treated under two different conditions and then their response to the condition is measured by one output variable. The cell populations in the two conditions are quite ...
3
votes
1answer
40 views

Is there any rule of thumb to delete a variable in a large data set?

I'm working with a large set as a project for the business analytic course with $10^5$ observations and 170+ variables, some of which come with a missing value proportion of larger than 20%, even more ...
3
votes
1answer
593 views

sample size too large? [duplicate]

I always thought larger sample sizes were better. Then I read something somewhere about how when sample sizes are larger, it's easier to find significant p-values when they're not really there (i.e., ...
1
vote
0answers
24 views

Joint significance testing w/ large samples

long time listener, first time caller. Unfortunately I can't show code as the computer the analysis is done on has rather tight security. I need to test for joint significance in a logit model that ...
3
votes
1answer
118 views

Robust estimates of the covariance matrix in the big data space

I am trying to compute the robust estimates of the covariance matrix (and also the mean) in the big data space. I am aware of FastMVE and FastMCD (Minimum Covariance Determinant and Minimum Volume ...
2
votes
1answer
123 views

How to fit OLS with many categorical levels, on more than one category

This question is not meant to be a software question, but I will illustrate the issue using R a bit. My Understanding of the Simple Case If I have a simple linear model with a categorical variable ...
0
votes
0answers
51 views

Quantifying the predictive ability of a model developed from a huge data set? (variation of bootstrapping?)

I have a statistical model with around 20 predictor variables, built on 90% of a dataset consisting of over 600k observations. The original developer held out 10% of the original dataset for the ...
5
votes
1answer
94 views

Wherefore Big Data?

It's been many years since I've done any statistics (or any serious math), but I do remember that the sampling error decreases more slowly for larger sample sizes (like n^-1/2, at least for some ...
0
votes
0answers
17 views

Difficulties of making inferences on large and high dimensional data

I've seen a couple of seemingly unrelated notes about working with large volumes of data and it struck me that I couldn't find much content on problems specific to statistical analysis of Big Data. ...
0
votes
0answers
237 views

Identifying multivariate outliers in a large sample with missing data, using SPSS

I'm a psychology PhD student doing analysis on a relatively large set of data, obtained via online surveys. The purpose of the study is largely to determine normative data for a population of adults, ...
0
votes
0answers
46 views

appropriate sample size to fit non-normal distribution parameters

I have a population that, in theory, I could determine exactly. I am not doing so, however, because on my poor home computer, it would be too computationally intensive. Each calculation takes 45-90 ...
1
vote
1answer
46 views

Big Data Regression Coefficient Estimation

I am working on a very large data set (n = 6.5 million) and I am trying to come up with a simple linear regression between two variables. I am working in R and using a monte carlo style simulation to ...
1
vote
1answer
113 views

Can binning a continuous predictor or DV variable improve large data sets fit?

I read that averaging and binning a continuous predictor variable is in general a bad idea because it's always better to fit the continuous relationship through splines, poly and all of that. Sure, I ...
1
vote
1answer
74 views

Is it possible to fit a logistic regression model to a dataset with categorical predictive variables with very high number of levels each?

I want to fit a model to a very large dataset, with a standard binary response variable and with 3 categorical predictor variables with 3000, 15 and 2 levels. Is there any inherent problem in this ...
0
votes
0answers
24 views

Linear-time approximation to kernel SVM?

Scaling kernel support vector machines to large datasets is a very challenging problem. For linear SVMs, PEGASOS is able to learn efficiently online, so training time scales linearly with the size of ...
6
votes
1answer
189 views

What are some interesting examples of wrong or crazy inferences being drawn from Big Data?

I'm interested in well-known examples of Big Data misinterpreted, poorly analysed, or wrongly employed to unscientific and incorrect ends. Would appreciate any examples or observations. Thanks a lot!
2
votes
2answers
150 views

Does the dataset size influence a machine learning algorithm?

So, imagine having access to sufficient data (millions of datapoints for training and testing) of sufficient quality. Please ignore concept drift for now and assume the data static and does not change ...
3
votes
0answers
86 views

Permutation for large sample sizes (and z-test)

I was trying to do a permutation test on a large amount of temperature observations in R. Approximately 1700 temperature observations (g1) from one place and 2 ...
0
votes
0answers
22 views

Big Data Use Cases

I am going to study about big data use cases in the real world. I want the stuff to be preferable in book format(not article or blog posts or website articles) and high level, industry based and ...
-1
votes
2answers
99 views

How do you draw a random sample of unique IDs in a large dataset?

Rookie here -- I have a a large data set with about 75,000 observations, and 2000 unique IDs. Therefore, each unique ID has about 37 observations. I'm trying to draw a random sample of unique IDs, say ...
1
vote
0answers
67 views

Calculation of an overall top X over several groups

I'm not a statistics expert, so I'm hoping someone here can lend me a hand. I've got a bunch of key-value pairs associated with a specific time range. Something like this: ...
2
votes
1answer
124 views

Comparing large categorical data sets with low or zero counts

I'm dealing with a biological feature which can be classified into $2^{20}$ categories. I also have two pretty large data sets of 1 and 3 million entries. Actually, only around 30 thousand categories ...
4
votes
2answers
110 views

How to reduce the dimension of $10^8$ vectors

I have $10^8$ vectors in $1000$ dimensions each. I would like to drastically reduce their dimensions. However PCA seems computationally infeasible. Are there near linear time methods to do ...
1
vote
1answer
94 views

Best metric for evaluation of mixture-of-Gaussian clusters on big-data

I have made a new algorithm that is specifically crafted for clustering very large datasets. In order to document it as a research paper, I have to choose one or two internal (no-label) cluster ...
1
vote
0answers
14 views

Sybil detection metric

I have a set of user data and I want to build some kind of metric to evaluate the probability of the user being a sybil (a "fake" account). But I have a very limited set of users who are sybils with ...