'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

0
votes
0answers
59 views

What skills are required to interpret the result of large scale statistical analyses? [on hold]

Suppose I have all the necessary skills to analyze a large data and I perform an analysis to I obtain results following the answer on a related question. Now I am overwhelmed by the task of ...
2
votes
0answers
19 views

Missing value imputation in huge dataset

I have a huge data (4M x 17) that has missing values. Two columns are categorical, rest all are numerical. Given the huge amount of data, running any imputation method runs forever. What should I do? ...
0
votes
0answers
23 views

A meaningful measure of synchronicity

I have a very large panel of data represented by the matrix elements $X_{t,k}\in\mathbb{R}$ where $t$ is a time-series index running in $1,..,T$ and $k$ is a cross-sectional index running in $1,...,N$....
7
votes
1answer
383 views

When have I to stop looking for a model?

I'm looking for a model between stockprices of energy and the weather. I have the price of the MWatt bought between the countries of Europe, and a lot of values on the weather (Grib files). Each hours ...
0
votes
2answers
44 views

Should using SMO classification in WEKA take so long with large dataset?

I have a dataset of 205 features and 238000 samples. It is a combined dataset of several subjects' data that I want to use for between-subject classification. I am using WEKA 3.8 with a 64-bit JVM ...
0
votes
0answers
6 views

Syntax for using window aggregate functions in Spark SQL? [migrated]

I am trying to use denseRank function in Apache Spark, as mentioned below: window = Window.partitionBy(data.userid).orderBy(data.event_date) ranked=data.select("userid","event_date",denseRank().over(...
3
votes
1answer
143 views

Things that I am not sure about “LASSO” regression method

I have read the chapters that are related to "LASSO" regression in: The elements of statistical learning (Tibshirani et al.) Statistical Learning with Sparsity: The Lasso and Generalizations. (...
0
votes
0answers
16 views

How to deal with missing data when calculating Information Gain

While working on a neural network for classification problem I'm dealing with huge number of possible features and information gain seems like a good way to narrow them down (there are hundreds of ...
3
votes
2answers
43 views

Density estimation for large dataset

I have a unidimensional data set with more than 1000000 observations. Assuming that those observations are independent realizations of the same random variable I need to estimate the underling ...
2
votes
1answer
31 views

What is your configuration to do statistical analysis on large data sets?

The stat packages which I've been using are not very well suited for working with large data sets, i.e. more than 1 billion records. They all tend to try loading all into memory, and install one one ...
4
votes
3answers
144 views

Real-world example on significance testing with large samples

The issues that arise when one uses statistical hypothesis testing framework with large samples are dissussed on CV (see e.g. this thread). The main problem discussed there is that in the real world ...
3
votes
0answers
24 views

How to check the assumptions behind an inference procedure, in the case of very large data sets

On this site it has been confirmed multiple times that, contrary to what is often heard, hypothesis tests don't have any issues with large sample sizes. As a matter of fact, the probability of Type 1 ...
0
votes
1answer
23 views

Reference for incremental sandwich covariance from biglm?

I am working on some similar methods to Lumley's biglm wrapper around Miller's AS274 algorithm, and I can't seem to find a reference for his incremental Huber/White ...
0
votes
0answers
48 views

Plot large high dimensional dataset

I have a high dimensional dataset (12 variables) with 42000 observations. I made a clustering and I found evidence that made me choose five clusters. So I want to plot my dataset with some tool to ...
2
votes
2answers
128 views

What to take in consideration when we use Bayesian Methods on Big Data problems?

I was reading the book Bayesian Methods for Hackers by Cameron Davidson-Pilon. He use PyMC for examples. As an experiment, I created a ...
1
vote
1answer
21 views

large sample and significance

I have a very large sample (790 million rows), and in this case, even small difference are determined to be significant, p <0.0001. The 95% C.I.s are very sharp. In this case, these tests (t-test, ...
0
votes
1answer
14 views

Chi squared test sample size

I'm trying to figure out if there's a difference across the main capitals in Europe and the voting preference of their inhabitants for extreme right political parties. So far I have categorized all ...
1
vote
0answers
20 views

Who does/can benefit from smart meters? [closed]

This is bit of a speculative question, and I apologize if this is the wrong forum (as is likely from the lack of applicable tags). I'm interested in theoretical benefits though, and this seems to be ...
1
vote
1answer
24 views

Trees of ensembles.

I have a large dataset (100k+), and it's growing everyday. I want to train it to predict a value (a regression problem). I've been finding that ensemble trees work the best for now, but in the ...
0
votes
0answers
21 views

Computing leave-one-out score of the linear regression for a large-scale regression

I heard that, for a linear regression, a leave-one-out cross validation score can be written in an explicit formula (using a matrix multiplication). (I browsed, e.g., http://robjhyndman.com/hyndsight/...
1
vote
0answers
8 views

Where can I find ressouce for big multi valued time series? [closed]

for my job I need to find timeseries datas to fit our algorithms. I have encoutered many links for Keogh archive and his work in general, but provided datas are "too clean" i.e datas are aligned, ...
1
vote
2answers
44 views

Is Naive Bayes suitable for large datasets with thousands of features?

I have a data set with 100 million rows and 15,000 categorical variables each with 0/1 values. My target variable is also a 0/1 binary variable. Is Naive Bayes suitable in terms of computational ...
2
votes
0answers
15 views

Mahalanobis distance for highly multivariate random variable

I have to compute the Mahalanobis distance for a $10^6$ dimensional multivariate random variable. What is the best (and fastest) way to do this? I am currently taking cholesky decomposition of the ...
0
votes
0answers
26 views

multi linear regression model. Modeling with a heavily skewed binary independent variable

Dataset and goal: One continuous measurement( to be modeled as a dependent variable) and four other measurements (one binary and the rest are category variables with multiple levels) to be modeled as ...
0
votes
0answers
14 views

What are the various ways to group a big time series (>2000) into few categories and apply one algorithm for each of these groups?

I have ~2000 time series to forecast. Do I need to be able to group them (for example, into 10) or find associations/dependencies between them (tests such as HHG, dCov etc.) so that I can use my ...
0
votes
0answers
39 views

How to compare groups when the control group is 100 times larger than the group in question?

I am working with NIS data which are basically a 20% sample of all in-hospital admissions in the US every year. I have small group of cases that carry diagnosis "X" in their record and overall the ...
1
vote
0answers
69 views

Server Requirements for machine learning on 300 million row database [closed]

I am trying to build a logistic regression model using Revolution R 64 bit. My data is 300 million rows by 12 columns. The data is stored on AWS Redshift, but takes about 6 hrs to import and save as ...
1
vote
0answers
19 views

How to store a large graph? [closed]

When I try to develop a spelling collector, I realize if I can include the word background (etc:before and after word) into the bayes calculation, may be could increase this spelling collector ...
0
votes
0answers
24 views

Help in understanding hashing for nearest neighbor search

Hashing is a technique for large- scale visual search and a variety of hashing-based method- s have been proposed Survey paper : Hashing for similarity search . The application of hashing to ...
0
votes
0answers
21 views

Hyperparameter optimization on large dataset

I have a huge dataset and want to carry out regression, such as gradient boosting. The problem is that the dataset is huge and hyperparameter optimization is computational expensive, especially I use ...
0
votes
0answers
60 views

How to visualize simplified dendrograms of many samples in R

I want to make sense out of my samples by visualizing my hierarchical clustering of a large dataset into a dendrogram in R. However as I have over 450k samples, I cannot make any sense out of the ...
0
votes
0answers
17 views

R model developing & validating - Open to Discussion [duplicate]

Throughout my R journey I have noticed the way we can use given data to develop and validate a model. Assume that you have given data for a problem train.csv test.csv Method A Combine train+...
2
votes
0answers
19 views

Aggregating covariances

Suppose I have 2 data sets too large to combine. Each data set has one row per user, $i$, and two columns $(x, y)$. Users can be in both data sets but have different column values, $(x_{i1}, y_{i1})$...
3
votes
1answer
71 views

Why does a large sample size cause a significant ANOVA F-test?

If I have a large sample size, e.g. 100,000 data points, I know that most significance tests are going to come back with a very small p-value unless the null hypothesis is "true on the nose." In other ...
1
vote
2answers
108 views

Cross Validation - purpose, need and utility [duplicate]

The question might sound like an old one but I haven't got satisfactory answers for a number of questions I have about CV. I looked at several questions on CV here, here, here and here and yet things ...
2
votes
0answers
46 views

how to model longitudinal big data?

Traditionally we use mixed model to model longitudinal data, i.e. data like: ...
0
votes
1answer
47 views

How to plot a line graph for very huge data file? [closed]

How to plot a graph for very huge data. I am data sets containing minimum of 300000 counts. I need to plot a graph. Is there any special package for it. Also, I need to insert secondary axis for the ...
0
votes
0answers
21 views

Which statistical assumptions are still important when fitting a GLM to > 1 million observations?

I have previously only fit GLM models to small/medium sized data (up to several thousand points, maybe tens of thousands). I always try to be meticulous about checking that GLM assumptions hold where ...
0
votes
0answers
14 views

Database and Data Waehouse design and processing in Big Data

With the growing popularity of Big Data and related tools, I am looking for books that specifically deal with implementing a warehouse in one of Big Data technologies, database and data-warehouse ...
0
votes
1answer
29 views

Binary random variable, big data frame: does my approach make sense?

I have a large data frame with about 1100 columns containing integers and about 30'000 rows. The last column contains a binary random variable which attains values 0 and 1. 30% of the data frame ...
3
votes
0answers
65 views

How to do dimensionality reduction on a huge data set?

I am working with fMRI data of ~1000 subject. Each subject has a feature vector of ~150 million dimension. So I can only keep the feature vectors of ~10 subjects in memory. What are some algorithms ...
0
votes
0answers
33 views

Using Mann-Whitney U instead of T-test in large sample

For my dissertation, I am examining racial differences in perceived choice and control and home and community-based services (HCBS) use in frail elders (n=659) who have moved from a nursing home into ...
0
votes
0answers
28 views

Best way to read HDFS data in HBase

I have a source where real time data is streaming into Flume and then Flume dumps it into HDFS. QUESTION: I believe that there are couple of ways to read HDFS file in HBase. Can anyone brief me about ...
0
votes
0answers
24 views

Evaluate fit of classification model in large sample

I have a large data sample with around n=300,000 observations. My dependent variable is categorical (0: no, 1: yes) and I have 10 independent variables which are categorized into 2 or 3. I run $\chi^...
0
votes
0answers
33 views

Time series pattern match in large datasets

I want to find in a time series pattern that look similar to a predifined one. At the moment I am using DTW to do this and it works well. But now I want to use it for larger datasets like finding a ...
1
vote
0answers
36 views

Cluster analysis of large, multiple answer dataset

I have to analyse data from a marketing study. I will use SPSS. The questionnaire will look like this: Q: Imagine Situation X. Select 1-3 Criteria from the list that best describe your feeling. ...
1
vote
1answer
83 views

SVM classifier - can I average multiple models?

I'm performing SVM classification on a relatively large data set (~1M rows, 4 variables). I want to assign a classification score to each row, not evaluate input parameters, so following the top ...
1
vote
0answers
59 views

Statistical testing in machine learning [closed]

I have a clarification. I am currently working as a data scientist for an IT company. I have experience in the area of NLP, ML and Big data for more than 5 years. I am from Bioinformatics background. ...