'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

1
vote
0answers
17 views

Sequential conditional simulation to avoid using a large covariance matrix

I would like to generate $S$ samples of a $T \cdot M$ dimensional vector, where $T$ is the number of time steps and $M$ the number of locations, i.e., the vector is a stack with $T$ values for ...
1
vote
1answer
25 views

Which statistical analysis to use to compare the level of similarity between two large samples?

I'm writing a small speech recognition prototype as my side project, which matches pre-recorded words of the speaker. So now I'm thinking of comparing two sets of data (outcome of FFT) which are two ...
0
votes
2answers
45 views

R: Multinomial Logistic Regression for health data

I am doing some data analysis on a fairly large health data set of patients with diagnoses and the respective procedures received for each event. I was asked to run a multinomial logistic regression ...
1
vote
1answer
21 views

Pattern Recognition - Visualizing Dense Data Points

I have a sample of around 5000 data (2D) points that are generated through a simulation of a cryptocurrency's mining events of following form. In column 2 one can see identical y-values with ...
1
vote
0answers
9 views

Taming large datasets. How to keep historical data with aggregation techniques?

I will be collecting huge amounts of computer performance data and the datasets will get large very quickly. This necessitates reducing the size of the data by compressing the historical data using ...
1
vote
2answers
26 views

Silhouette clustering index in practice

I don't have much experience with data analysis algorithms (data mining, machine learning, if you like) and I'm interested if some could share their experience with practical usage of Silhouette in ...
0
votes
0answers
18 views

Multivariate discretization method / library for huge data

Does anyone know any multivariate discretization method that can be used for large amounts of data. A library / Python library would be awesome but algorithms would also do. Also I'm not sure if ...
0
votes
0answers
31 views

What is the best way to test normality for a very large sample size 15000? [duplicate]

Also do I need to test the normality for the categorical data? Thanks
2
votes
0answers
25 views

Classification and regression tree (CART) on large data set

I am trying to approximate a multivariate function $y = f(x_1, ...x_n)$, which I have reason to believe will be well approximated by a classification and regression tree. Some of the variables are ...
0
votes
2answers
33 views

Increasing the power by dropping points: can I do it?

I am repeating a test on a large amount of data and FDR-correcting the p-values afterwards for multiple testing. Yet I still do not have enough power. However, I feel like it is not necessary to test ...
0
votes
0answers
36 views

Visualize multivariate data in Excel

Here is an example of the data I want to visualize either as stacked bar or as scatter plot. ...
1
vote
1answer
62 views

Machine learning tutorials / examples on data sets larger than a terabyte

I am trying to gather a list of practical ML examples / tutorials on more than a terabyte of data. I'm particularly interested in feature extraction from large data sets that involves aggregation (the ...
1
vote
0answers
31 views

Need an effective way to show distribution changes over time and outlier reoccurence

Does anyone have suggestions on the best way to approach this problem? I have a large dataset (over 200k+ per day) in a MySQL database, that consists of a single record per user per day with a ...
0
votes
1answer
35 views

Choose software tool for basic analysis [closed]

(I am pretty novice in data analysis) So, I have a set of elements (~105). All elements splitted for some classes of disjoint sets. Actually, there are two systems of sets. ...
0
votes
0answers
21 views

benefices of big data on machine learning methodologies

I know that there are a number of predictive models (generized linear ones, trees, neural network, support vector machines, knn, Naive Bayes, ...) that have been proposed to perform various analytical ...
0
votes
2answers
62 views

Using k-means for reducing the size of the training set of a Kernel SVM

I have a classification problem with the following characteristics: a few million data points around one hundred features non-linearly separable Training a SVM with an RBF Kernel is not feasible ...
1
vote
2answers
57 views

Non-parametric tests in big data scenario

Suppose I have two populations A and B , with sizes $n_1$ and $n_2$ respectively, where both $n_1$ and $n_2$ are large (say, above 500). I want to test that the values $x_1, \dots, x_{n_1}$ of A ...
0
votes
1answer
44 views

Best ways to model 'big data' given limited computing resources

Suppose I have a large data set(10 GB), with a response variable and multiple independent variables. What is the best way to utilize the data to build a model on? If the full data set includes 10 ...
0
votes
0answers
22 views

Uncertain about research strategy

I am currently busy with defining my methodology/research design. What I am trying to research is why consumers avoid the first retailer (the cheapest retailer in the shopbot) and what factors drive ...
1
vote
0answers
19 views

Unsure which tests to use now that my continuous symptom scale has been replaced with its underlying subscales

I've been working with a large data set in SPSS, mostly focused on the symptom scales of one developmental disorder. I have data from two separate time points, a few years apart. So for example I've ...
0
votes
0answers
33 views

Associating non-linear three-time-point change with a continuous variable

I would be incredibly grateful for help or advice regarding my following project: I have 3 time points (0, 30, 120 min) and complete data for about $n=500$ subjects for a continuous variable $M$. ...
12
votes
4answers
392 views

Confidence intervals when the sample size is very large

My question could be rephrased as "how to assess a sampling error using big data", especially for a journal publication. Here is an example to illustrate a challenge. From a very large dataset ...
0
votes
0answers
36 views

Variable importance in classification?

For example: I have 100 books with 1000 words each. They belong to different classes (comedy,drama,...). Each class consist of 15 different books. When i do TDIDF (term frequency - inverse document ...
0
votes
0answers
32 views

Checking assumptions of a large sample - extremely confused

I'm working with a sample size of over 2000 and I have become extremely confused at the first hurdle... I plan to run three linear regressions and one logistic regression. If I was working with a ...
2
votes
0answers
28 views

Instances of sparse covariance matrices

I am trying to find large datasets with inherently sparse covariance matrices, to be tested with our algorithm. Basically, we will take the sample covariance matrix and enforce some structured ...
0
votes
0answers
43 views

Large samples and the p-values almost zero in GEE models

I’m trying to find correlations between some environmental variables and a binary variable, berried/not berried females (rock lobster). I have 20 years of data that were sampled on a daily basis, ...
1
vote
1answer
68 views

Practical collaborative filtering application for large database

I’m designing an item-based collaborative filtering for a large database with over 100,000 items. My question is how the whole process works in practice since the algorithm takes a long time to ...
4
votes
1answer
41 views

Distributed Datasets and MLE

Suppose I have a very large dataset of size N, evenly distributed over M computers so that each computer has N/M data points. Suppose I want to fit a model using MLE that requires an iterative method. ...
2
votes
1answer
134 views

Big data database + software for advanced statistical analysis?

I need to run some statistical hypothesis testing, Anova, student's, least square fit, median, data mining, clustering... on a very large quantity of distributed data. (>100TB, Maybe columnar or ...
3
votes
1answer
100 views

Should I use Mean Square Error or Classification Rate?

I am a self-taught person and I would like your help. I am learning about predictive modeling in general, and I'm also trying to do predictive modeling for a specific problem. I am exploring ...
4
votes
2answers
192 views

Can I use likelihood-ratio test to compare two samples drawn from power-law distributions?

I have to compare two large samples ($N = 10^{6}$) of discrete data drawn from power-law distributions to assess whether they are significantly different. I can't do that by means of a two-sample ...
1
vote
0answers
31 views

sklearn - Multinomial Naive Bayes (data too big???!!!)

I wanted to really understand the rationale behind the following code as written in python sklearn's manual partial_fit(X, y, classes=None, sample_weight=None) when the data is too big to fit in ...
0
votes
1answer
46 views

MARS for big data

Is there a way in which MARS/EARTH can work with ff objects or big data? I need an additive model for a sample of size 20000 and around 100 variables. I can not work with that dimension with a RAM ...
1
vote
0answers
44 views

Feature space reduction for tag prediction

[x-post] from stackoverflow. I am writing a ML module (python) to predict tags for a stackoverflow question (tag + body). My corpus is of around 5 million questions with title, body and tags for ...
0
votes
1answer
99 views

How to apply an R prediction model to very big data from SQL database in parallel.

I dont need to load the entire dataset into memory. In fact I only need 1 row at a time to apply a trained model, get the predicted response and put that response somewhere, possibly back into another ...
3
votes
1answer
62 views

Is an F-test for equality of variance appropriate for a very large dataset?

I have a dataset with about 500,000 subjects and I am trying to establish whether the variance is equal. I first performed an F-test but then I realised the data is slightly skewed with kurtosis. So ...
4
votes
3answers
194 views

Why do irrelevant regressors become statistically significant in large samples?

I am trying to better understand statistical significance, effect sizes and the like. I have a perception (perhaps its wrong) that even irrelevant regressors often become statistically significant in ...
4
votes
1answer
72 views

Propensity score matching with large data

I have a large healthcare claims database with 1.6 million subjects and I'm interested in doing a cohort study with propensity score matching. I have produced my propensity score with a logistic ...
0
votes
0answers
28 views

Enterprise Use of SAS & SPSS and Impact of Open Source Platforms (R & Python) [duplicate]

I'm curious to hear from people that use SAS & SPSS why a lot of the Fortune 100/500 enterprises still use SAS & SPSS despite the high cost of the license. Also would love to understand what ...
1
vote
1answer
115 views

Classification problem-Big Data and simple decision rules: logit regression, LDA, random forest, cond. trees, or something else?

This is a big data question from someone who is more accustomed to small data. I would like to develop some classification "rules of thumb," that is, some simple decision rules or a decision tree ...
3
votes
2answers
109 views

Collinearity in multivariate regression with huge amounts of data

Take the following example. I wish to predict physical performance as a function of height and weight. I already know weight negatively affects performance. Height also negatively affects performance, ...
0
votes
1answer
152 views

Large data variable selection

I'm looking for some methods of variable selection on large datasets.The number of variables are around 30-40, but the number of observations is quite large (around 36000000) Any methods which I ...
0
votes
0answers
69 views

Diference between glm and bigglm estimates

How does bigglm function in biglm package work for logistic regression? I thought that it is not possible to calculate LR on chunks of data and then merge results. Will glm and bigglm yield ...
1
vote
2answers
208 views

Compare two distributions of large sizes and unequal variances where one distribution is heavily skewed

My data is from cells that are treated under two different conditions and then their response to the condition is measured by one output variable. The cell populations in the two conditions are quite ...
3
votes
1answer
49 views

Is there any rule of thumb to delete a variable in a large data set?

I'm working with a large set as a project for the business analytic course with $10^5$ observations and 170+ variables, some of which come with a missing value proportion of larger than 20%, even more ...
3
votes
1answer
1k views

sample size too large? [duplicate]

I always thought larger sample sizes were better. Then I read something somewhere about how when sample sizes are larger, it's easier to find significant p-values when they're not really there (i.e., ...
1
vote
0answers
28 views

Joint significance testing w/ large samples

long time listener, first time caller. Unfortunately I can't show code as the computer the analysis is done on has rather tight security. I need to test for joint significance in a logit model that ...
3
votes
1answer
179 views

Robust estimates of the covariance matrix in the big data space

I am trying to compute the robust estimates of the covariance matrix (and also the mean) in the big data space. I am aware of FastMVE and FastMCD (Minimum Covariance Determinant and Minimum Volume ...
2
votes
1answer
196 views

How to fit OLS with many categorical levels, on more than one category

This question is not meant to be a software question, but I will illustrate the issue using R a bit. My Understanding of the Simple Case If I have a simple linear model with a categorical variable ...
0
votes
0answers
52 views

Quantifying the predictive ability of a model developed from a huge data set? (variation of bootstrapping?)

I have a statistical model with around 20 predictor variables, built on 90% of a dataset consisting of over 600k observations. The original developer held out 10% of the original dataset for the ...