'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

2
votes
0answers
67 views

Structuring Many-Factor Data for Linear Regression in R

I have a fairly large dataset of the following form, and I want to run a linear regression returning coefficients for each factor: ...
0
votes
0answers
7 views

Ranking of significant (and insignificant) coefficients of regional dummies

I have multiple regression of dependent variable on some regional dummies. I see in a paper (Oswald 2010) that the author rank estimated coefficients of the state dummies. My sample size is about ...
0
votes
0answers
10 views

Core vector machine implementation

I came across the following article : http://www.jmlr.org/papers/volume6/tsang05a/tsang05a.pdf, Core Vector Machines: Fast SVM Training on Very Large Data Sets. The approach looks very promising, ...
2
votes
2answers
289 views

Fisher's test with large data

I want to compare the proportions in two samples, that can be organized as P NP SAMPLE_A 3,129,548 427,953 SAMPLE_B 2,930,639 407,353 If I ...
0
votes
0answers
3 views

Non-spreadsheet-esque Data Import and Organization [migrated]

Basic Problem I have data of the form doc id 1 var1 A var2 B ... varStar 453 varStar 3432 varStar 32 ... varN doc var1 A var2 B (And so on) Where ...
2
votes
0answers
30 views

Survival analysis with Frailty on large dataset

I am trying to fit a survival analysis in R with non-recurrent events and time-varying coefficients. The baseline distribution is exponential or Weibull and the ...
1
vote
1answer
27 views

How to apply distance-based clustering or dimensionality reduction for too many samples

I have a dataset with 200K samples (cases) and 30 variables. Every distance-based method for clustering or dimension reduction technique that I use, such as DBSCAN, Hierarchical Clustering, LLE, ...
0
votes
0answers
31 views

Training set selection

I have the following question for a project I'm working on. I am trying to find the best strategy to select the best training set in a dataset. I have a dataset with a few billions rows. I am trying ...
1
vote
0answers
25 views

Sequential conditional simulation to avoid using a large covariance matrix

I would like to generate $S$ samples of a $T \cdot M$ dimensional vector, where $T$ is the number of time steps and $M$ the number of locations, i.e., the vector is a stack with $T$ values for ...
1
vote
1answer
40 views

Which statistical analysis to use to compare the level of similarity between two large samples?

I'm writing a small speech recognition prototype as my side project, which matches pre-recorded words of the speaker. So now I'm thinking of comparing two sets of data (outcome of FFT) which are two ...
0
votes
2answers
50 views

R: Multinomial Logistic Regression for health data

I am doing some data analysis on a fairly large health data set of patients with diagnoses and the respective procedures received for each event. I was asked to run a multinomial logistic regression ...
1
vote
1answer
21 views

Pattern Recognition - Visualizing Dense Data Points

I have a sample of around 5000 data (2D) points that are generated through a simulation of a cryptocurrency's mining events of following form. In column 2 one can see identical y-values with ...
1
vote
0answers
13 views

Taming large datasets. How to keep historical data with aggregation techniques?

I will be collecting huge amounts of computer performance data and the datasets will get large very quickly. This necessitates reducing the size of the data by compressing the historical data using ...
1
vote
2answers
41 views

Silhouette clustering index in practice

I don't have much experience with data analysis algorithms (data mining, machine learning, if you like) and I'm interested if some could share their experience with practical usage of Silhouette in ...
0
votes
0answers
21 views

Multivariate discretization method / library for huge data

Does anyone know any multivariate discretization method that can be used for large amounts of data. A library / Python library would be awesome but algorithms would also do. Also I'm not sure if ...
0
votes
0answers
34 views

What is the best way to test normality for a very large sample size 15000? [duplicate]

Also do I need to test the normality for the categorical data? Thanks
2
votes
0answers
32 views

Classification and regression tree (CART) on large data set

I am trying to approximate a multivariate function $y = f(x_1, ...x_n)$, which I have reason to believe will be well approximated by a classification and regression tree. Some of the variables are ...
0
votes
2answers
33 views

Increasing the power by dropping points: can I do it?

I am repeating a test on a large amount of data and FDR-correcting the p-values afterwards for multiple testing. Yet I still do not have enough power. However, I feel like it is not necessary to test ...
0
votes
0answers
43 views

Visualize multivariate data in Excel

Here is an example of the data I want to visualize either as stacked bar or as scatter plot. ...
1
vote
1answer
69 views

Machine learning tutorials / examples on data sets larger than a terabyte

I am trying to gather a list of practical ML examples / tutorials on more than a terabyte of data. I'm particularly interested in feature extraction from large data sets that involves aggregation (the ...
1
vote
0answers
49 views

Need an effective way to show distribution changes over time and outlier reoccurence

Does anyone have suggestions on the best way to approach this problem? I have a large dataset (over 200k+ per day) in a MySQL database, that consists of a single record per user per day with a ...
0
votes
1answer
37 views

Choose software tool for basic analysis [closed]

(I am pretty novice in data analysis) So, I have a set of elements (~105). All elements splitted for some classes of disjoint sets. Actually, there are two systems of sets. ...
0
votes
0answers
21 views

benefices of big data on machine learning methodologies

I know that there are a number of predictive models (generized linear ones, trees, neural network, support vector machines, knn, Naive Bayes, ...) that have been proposed to perform various analytical ...
0
votes
2answers
65 views

Using k-means for reducing the size of the training set of a Kernel SVM

I have a classification problem with the following characteristics: a few million data points around one hundred features non-linearly separable Training a SVM with an RBF Kernel is not feasible ...
1
vote
2answers
64 views

Non-parametric tests in big data scenario

Suppose I have two populations A and B , with sizes $n_1$ and $n_2$ respectively, where both $n_1$ and $n_2$ are large (say, above 500). I want to test that the values $x_1, \dots, x_{n_1}$ of A ...
0
votes
1answer
44 views

Best ways to model 'big data' given limited computing resources

Suppose I have a large data set(10 GB), with a response variable and multiple independent variables. What is the best way to utilize the data to build a model on? If the full data set includes 10 ...
0
votes
0answers
22 views

Uncertain about research strategy

I am currently busy with defining my methodology/research design. What I am trying to research is why consumers avoid the first retailer (the cheapest retailer in the shopbot) and what factors drive ...
1
vote
0answers
19 views

Unsure which tests to use now that my continuous symptom scale has been replaced with its underlying subscales

I've been working with a large data set in SPSS, mostly focused on the symptom scales of one developmental disorder. I have data from two separate time points, a few years apart. So for example I've ...
0
votes
0answers
33 views

Associating non-linear three-time-point change with a continuous variable

I would be incredibly grateful for help or advice regarding my following project: I have 3 time points (0, 30, 120 min) and complete data for about $n=500$ subjects for a continuous variable $M$. ...
12
votes
4answers
408 views

Confidence intervals when the sample size is very large

My question could be rephrased as "how to assess a sampling error using big data", especially for a journal publication. Here is an example to illustrate a challenge. From a very large dataset ...
0
votes
0answers
41 views

Variable importance in classification?

For example: I have 100 books with 1000 words each. They belong to different classes (comedy,drama,...). Each class consist of 15 different books. When i do TDIDF (term frequency - inverse document ...
0
votes
0answers
32 views

Checking assumptions of a large sample - extremely confused

I'm working with a sample size of over 2000 and I have become extremely confused at the first hurdle... I plan to run three linear regressions and one logistic regression. If I was working with a ...
2
votes
0answers
29 views

Instances of sparse covariance matrices

I am trying to find large datasets with inherently sparse covariance matrices, to be tested with our algorithm. Basically, we will take the sample covariance matrix and enforce some structured ...
0
votes
0answers
43 views

Large samples and the p-values almost zero in GEE models

I’m trying to find correlations between some environmental variables and a binary variable, berried/not berried females (rock lobster). I have 20 years of data that were sampled on a daily basis, ...
1
vote
1answer
75 views

Practical collaborative filtering application for large database

I’m designing an item-based collaborative filtering for a large database with over 100,000 items. My question is how the whole process works in practice since the algorithm takes a long time to ...
4
votes
1answer
41 views

Distributed Datasets and MLE

Suppose I have a very large dataset of size N, evenly distributed over M computers so that each computer has N/M data points. Suppose I want to fit a model using MLE that requires an iterative method. ...
2
votes
1answer
150 views

Big data database + software for advanced statistical analysis?

I need to run some statistical hypothesis testing, Anova, student's, least square fit, median, data mining, clustering... on a very large quantity of distributed data. (>100TB, Maybe columnar or ...
3
votes
1answer
108 views

Should I use Mean Square Error or Classification Rate?

I am a self-taught person and I would like your help. I am learning about predictive modeling in general, and I'm also trying to do predictive modeling for a specific problem. I am exploring ...
4
votes
2answers
213 views

Can I use likelihood-ratio test to compare two samples drawn from power-law distributions?

I have to compare two large samples ($N = 10^{6}$) of discrete data drawn from power-law distributions to assess whether they are significantly different. I can't do that by means of a two-sample ...
1
vote
0answers
34 views

sklearn - Multinomial Naive Bayes (data too big???!!!)

I wanted to really understand the rationale behind the following code as written in python sklearn's manual partial_fit(X, y, classes=None, sample_weight=None) when the data is too big to fit in ...
0
votes
1answer
50 views

MARS for big data

Is there a way in which MARS/EARTH can work with ff objects or big data? I need an additive model for a sample of size 20000 and around 100 variables. I can not work with that dimension with a RAM ...
1
vote
0answers
45 views

Feature space reduction for tag prediction

[x-post] from stackoverflow. I am writing a ML module (python) to predict tags for a stackoverflow question (tag + body). My corpus is of around 5 million questions with title, body and tags for ...
0
votes
1answer
119 views

How to apply an R prediction model to very big data from SQL database in parallel.

I dont need to load the entire dataset into memory. In fact I only need 1 row at a time to apply a trained model, get the predicted response and put that response somewhere, possibly back into another ...
3
votes
1answer
66 views

Is an F-test for equality of variance appropriate for a very large dataset?

I have a dataset with about 500,000 subjects and I am trying to establish whether the variance is equal. I first performed an F-test but then I realised the data is slightly skewed with kurtosis. So ...
4
votes
3answers
203 views

Why do irrelevant regressors become statistically significant in large samples?

I am trying to better understand statistical significance, effect sizes and the like. I have a perception (perhaps its wrong) that even irrelevant regressors often become statistically significant in ...
4
votes
1answer
77 views

Propensity score matching with large data

I have a large healthcare claims database with 1.6 million subjects and I'm interested in doing a cohort study with propensity score matching. I have produced my propensity score with a logistic ...
0
votes
0answers
28 views

Enterprise Use of SAS & SPSS and Impact of Open Source Platforms (R & Python) [duplicate]

I'm curious to hear from people that use SAS & SPSS why a lot of the Fortune 100/500 enterprises still use SAS & SPSS despite the high cost of the license. Also would love to understand what ...
1
vote
1answer
121 views

Classification problem-Big Data and simple decision rules: logit regression, LDA, random forest, cond. trees, or something else?

This is a big data question from someone who is more accustomed to small data. I would like to develop some classification "rules of thumb," that is, some simple decision rules or a decision tree ...
3
votes
2answers
118 views

Collinearity in multivariate regression with huge amounts of data

Take the following example. I wish to predict physical performance as a function of height and weight. I already know weight negatively affects performance. Height also negatively affects performance, ...