'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

1
vote
0answers
12 views

Need an effective way to show distribution changes over time and outlier reoccurence

Does anyone have suggestions on the best way to approach this problem? I have a large dataset (over 200k+ per day) in a MySQL database, that consists of a single record per user per day with a ...
0
votes
1answer
26 views

Choose software tool for basic analysis [closed]

(I am pretty novice in data analysis) So, I have a set of elements (~105). All elements splitted for some classes of disjoint sets. Actually, there are two systems of sets. ...
0
votes
0answers
19 views

benefices of big data on machine learning methodologies

I know that there are a number of predictive models (generized linear ones, trees, neural network, support vector machines, knn, Naive Bayes, ...) that have been proposed to perform various analytical ...
0
votes
2answers
41 views

Using k-means for reducing the size of the training set of a Kernel SVM

I have a classification problem with the following characteristics: a few million data points around one hundred features non-linearly separable Training a SVM with an RBF Kernel is not feasible ...
2
votes
2answers
39 views

Non-parametric tests in big data scenario

Suppose I have two populations A and B , with sizes $n_1$ and $n_2$ respectively, where both $n_1$ and $n_2$ are large (say, above 500). I want to test that the values $x_1, \dots, x_{n_1}$ of A ...
0
votes
1answer
39 views

Best ways to model 'big data' given limited computing resources

Suppose I have a large data set(10 GB), with a response variable and multiple independent variables. What is the best way to utilize the data to build a model on? If the full data set includes 10 ...
0
votes
0answers
22 views

Uncertain about research strategy

I am currently busy with defining my methodology/research design. What I am trying to research is why consumers avoid the first retailer (the cheapest retailer in the shopbot) and what factors drive ...
1
vote
0answers
18 views

Unsure which tests to use now that my continuous symptom scale has been replaced with its underlying subscales

I've been working with a large data set in SPSS, mostly focused on the symptom scales of one developmental disorder. I have data from two separate time points, a few years apart. So for example I've ...
0
votes
0answers
31 views

Associating non-linear three-time-point change with a continuous variable

I would be incredibly grateful for help or advice regarding my following project: I have 3 time points (0, 30, 120 min) and complete data for about $n=500$ subjects for a continuous variable $M$. ...
12
votes
4answers
339 views

Confidence intervals when the sample size is very large

My question could be rephrased as "how to assess a sampling error using big data", especially for a journal publication. Here is an example to illustrate a challenge. From a very large dataset ...
0
votes
0answers
22 views

Variable importance in classification?

For example: I have 100 books with 1000 words each. They belong to different classes (comedy,drama,...). Each class consist of 15 different books. When i do TDIDF (term frequency - inverse document ...
0
votes
0answers
27 views

Checking assumptions of a large sample - extremely confused

I'm working with a sample size of over 2000 and I have become extremely confused at the first hurdle... I plan to run three linear regressions and one logistic regression. If I was working with a ...
2
votes
0answers
25 views

Instances of sparse covariance matrices

I am trying to find large datasets with inherently sparse covariance matrices, to be tested with our algorithm. Basically, we will take the sample covariance matrix and enforce some structured ...
0
votes
0answers
28 views

Large samples and the p-values almost zero in GEE models

I’m trying to find correlations between some environmental variables and a binary variable, berried/not berried females (rock lobster). I have 20 years of data that were sampled on a daily basis, ...
1
vote
1answer
49 views

Practical collaborative filtering application for large database

I’m designing an item-based collaborative filtering for a large database with over 100,000 items. My question is how the whole process works in practice since the algorithm takes a long time to ...
4
votes
1answer
40 views

Distributed Datasets and MLE

Suppose I have a very large dataset of size N, evenly distributed over M computers so that each computer has N/M data points. Suppose I want to fit a model using MLE that requires an iterative method. ...
2
votes
1answer
81 views

Big data database + software for advanced statistical analysis?

I need to run some statistical hypothesis testing, Anova, student's, least square fit, median, data mining, clustering... on a very large quantity of distributed data. (>100TB, Maybe columnar or ...
3
votes
1answer
74 views

Should I use Mean Square Error or Classification Rate?

I am a self-taught person and I would like your help. I am learning about predictive modeling in general, and I'm also trying to do predictive modeling for a specific problem. I am exploring ...
4
votes
2answers
132 views

Can I use likelihood-ratio test to compare two samples drawn from power-law distributions?

I have to compare two large samples ($N = 10^{6}$) of discrete data drawn from power-law distributions to assess whether they are significantly different. I can't do that by means of a two-sample ...
1
vote
0answers
22 views

sklearn - Multinomial Naive Bayes (data too big???!!!)

I wanted to really understand the rationale behind the following code as written in python sklearn's manual partial_fit(X, y, classes=None, sample_weight=None) when the data is too big to fit in ...
0
votes
1answer
38 views

MARS for big data

Is there a way in which MARS/EARTH can work with ff objects or big data? I need an additive model for a sample of size 20000 and around 100 variables. I can not work with that dimension with a RAM ...
1
vote
0answers
40 views

Feature space reduction for tag prediction

[x-post] from stackoverflow. I am writing a ML module (python) to predict tags for a stackoverflow question (tag + body). My corpus is of around 5 million questions with title, body and tags for ...
0
votes
1answer
69 views

How to apply an R prediction model to very big data from SQL database in parallel.

I dont need to load the entire dataset into memory. In fact I only need 1 row at a time to apply a trained model, get the predicted response and put that response somewhere, possibly back into another ...
3
votes
1answer
52 views

Is an F-test for equality of variance appropriate for a very large dataset?

I have a dataset with about 500,000 subjects and I am trying to establish whether the variance is equal. I first performed an F-test but then I realised the data is slightly skewed with kurtosis. So ...
4
votes
3answers
163 views

Why do irrelevant regressors become statistically significant in large samples?

I am trying to better understand statistical significance, effect sizes and the like. I have a perception (perhaps its wrong) that even irrelevant regressors often become statistically significant in ...
3
votes
0answers
43 views

Propensity score matching with large data

I have a large healthcare claims database with 1.6 million subjects and I'm interested in doing a cohort study with propensity score matching. I have produced my propensity score with a logistic ...
0
votes
0answers
26 views

Enterprise Use of SAS & SPSS and Impact of Open Source Platforms (R & Python) [duplicate]

I'm curious to hear from people that use SAS & SPSS why a lot of the Fortune 100/500 enterprises still use SAS & SPSS despite the high cost of the license. Also would love to understand what ...
1
vote
1answer
93 views

Classification problem-Big Data and simple decision rules: logit regression, LDA, random forest, cond. trees, or something else?

This is a big data question from someone who is more accustomed to small data. I would like to develop some classification "rules of thumb," that is, some simple decision rules or a decision tree ...
3
votes
2answers
88 views

Collinearity in multivariate regression with huge amounts of data

Take the following example. I wish to predict physical performance as a function of height and weight. I already know weight negatively affects performance. Height also negatively affects performance, ...
0
votes
1answer
114 views

Large data variable selection

I'm looking for some methods of variable selection on large datasets.The number of variables are around 30-40, but the number of observations is quite large (around 36000000) Any methods which I ...
0
votes
0answers
45 views

Diference between glm and bigglm estimates

How does bigglm function in biglm package work for logistic regression? I thought that it is not possible to calculate LR on chunks of data and then merge results. Will glm and bigglm yield ...
0
votes
1answer
88 views

Compare two distributions of large sizes and unequal variances where one distribution is heavily skewed

My data is from cells that are treated under two different conditions and then their response to the condition is measured by one output variable. The cell populations in the two conditions are quite ...
3
votes
1answer
41 views

Is there any rule of thumb to delete a variable in a large data set?

I'm working with a large set as a project for the business analytic course with $10^5$ observations and 170+ variables, some of which come with a missing value proportion of larger than 20%, even more ...
3
votes
1answer
843 views

sample size too large? [duplicate]

I always thought larger sample sizes were better. Then I read something somewhere about how when sample sizes are larger, it's easier to find significant p-values when they're not really there (i.e., ...
1
vote
0answers
26 views

Joint significance testing w/ large samples

long time listener, first time caller. Unfortunately I can't show code as the computer the analysis is done on has rather tight security. I need to test for joint significance in a logit model that ...
3
votes
1answer
134 views

Robust estimates of the covariance matrix in the big data space

I am trying to compute the robust estimates of the covariance matrix (and also the mean) in the big data space. I am aware of FastMVE and FastMCD (Minimum Covariance Determinant and Minimum Volume ...
2
votes
1answer
140 views

How to fit OLS with many categorical levels, on more than one category

This question is not meant to be a software question, but I will illustrate the issue using R a bit. My Understanding of the Simple Case If I have a simple linear model with a categorical variable ...
0
votes
0answers
51 views

Quantifying the predictive ability of a model developed from a huge data set? (variation of bootstrapping?)

I have a statistical model with around 20 predictor variables, built on 90% of a dataset consisting of over 600k observations. The original developer held out 10% of the original dataset for the ...
5
votes
1answer
100 views

Wherefore Big Data?

It's been many years since I've done any statistics (or any serious math), but I do remember that the sampling error decreases more slowly for larger sample sizes (like n^-1/2, at least for some ...
0
votes
0answers
18 views

Difficulties of making inferences on large and high dimensional data

I've seen a couple of seemingly unrelated notes about working with large volumes of data and it struck me that I couldn't find much content on problems specific to statistical analysis of Big Data. ...
0
votes
0answers
344 views

Identifying multivariate outliers in a large sample with missing data, using SPSS

I'm a psychology PhD student doing analysis on a relatively large set of data, obtained via online surveys. The purpose of the study is largely to determine normative data for a population of adults, ...
0
votes
0answers
48 views

appropriate sample size to fit non-normal distribution parameters

I have a population that, in theory, I could determine exactly. I am not doing so, however, because on my poor home computer, it would be too computationally intensive. Each calculation takes 45-90 ...
1
vote
1answer
46 views

Big Data Regression Coefficient Estimation

I am working on a very large data set (n = 6.5 million) and I am trying to come up with a simple linear regression between two variables. I am working in R and using a monte carlo style simulation to ...
1
vote
1answer
137 views

Can binning a continuous predictor or DV variable improve large data sets fit?

I read that averaging and binning a continuous predictor variable is in general a bad idea because it's always better to fit the continuous relationship through splines, poly and all of that. Sure, I ...
1
vote
1answer
77 views

Is it possible to fit a logistic regression model to a dataset with categorical predictive variables with very high number of levels each?

I want to fit a model to a very large dataset, with a standard binary response variable and with 3 categorical predictor variables with 3000, 15 and 2 levels. Is there any inherent problem in this ...
0
votes
0answers
26 views

Linear-time approximation to kernel SVM?

Scaling kernel support vector machines to large datasets is a very challenging problem. For linear SVMs, PEGASOS is able to learn efficiently online, so training time scales linearly with the size of ...
6
votes
1answer
197 views

What are some interesting examples of wrong or crazy inferences being drawn from Big Data?

I'm interested in well-known examples of Big Data misinterpreted, poorly analysed, or wrongly employed to unscientific and incorrect ends. Would appreciate any examples or observations. Thanks a lot!
2
votes
2answers
173 views

Does the dataset size influence a machine learning algorithm?

So, imagine having access to sufficient data (millions of datapoints for training and testing) of sufficient quality. Please ignore concept drift for now and assume the data static and does not change ...
3
votes
0answers
92 views

Permutation for large sample sizes (and z-test)

I was trying to do a permutation test on a large amount of temperature observations in R. Approximately 1700 temperature observations (g1) from one place and 2 ...
0
votes
0answers
23 views

Big Data Use Cases

I am going to study about big data use cases in the real world. I want the stuff to be preferable in book format(not article or blog posts or website articles) and high level, industry based and ...