'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

0
votes
0answers
40 views

Plot large high dimensional dataset

I have a high dimensional dataset (12 variables) with 42000 observations. I made a clustering and I found evidence that made me choose five clusters. So I want to plot my dataset with some tool to ...
2
votes
2answers
65 views

What to take in consideration when we use Bayesian Methods on Big Data problems?

I was reading the book Bayesian Methods for Hackers by Cameron Davidson-Pilon. He use PyMC for examples. As an experiment, I created a ...
1
vote
1answer
17 views

large sample and significance

I have a very large sample (790 million rows), and in this case, even small difference are determined to be significant, p <0.0001. The 95% C.I.s are very sharp. In this case, these tests (t-test, ...
0
votes
1answer
12 views

Chi squared test sample size

I'm trying to figure out if there's a difference across the main capitals in Europe and the voting preference of their inhabitants for extreme right political parties. So far I have categorized all ...
1
vote
0answers
18 views

Who does/can benefit from smart meters? [closed]

This is bit of a speculative question, and I apologize if this is the wrong forum (as is likely from the lack of applicable tags). I'm interested in theoretical benefits though, and this seems to be ...
1
vote
1answer
21 views

Trees of ensembles.

I have a large dataset (100k+), and it's growing everyday. I want to train it to predict a value (a regression problem). I've been finding that ensemble trees work the best for now, but in the ...
0
votes
0answers
20 views

Computing leave-one-out score of the linear regression for a large-scale regression

I heard that, for a linear regression, a leave-one-out cross validation score can be written in an explicit formula (using a matrix multiplication). (I browsed, e.g., ...
1
vote
0answers
7 views

Where can I find ressouce for big multi valued time series? [closed]

for my job I need to find timeseries datas to fit our algorithms. I have encoutered many links for Keogh archive and his work in general, but provided datas are "too clean" i.e datas are aligned, ...
1
vote
2answers
35 views

Is Naive Bayes suitable for large datasets with thousands of features?

I have a data set with 100 million rows and 15,000 categorical variables each with 0/1 values. My target variable is also a 0/1 binary variable. Is Naive Bayes suitable in terms of computational ...
2
votes
0answers
14 views

Mahalanobis distance for highly multivariate random variable

I have to compute the Mahalanobis distance for a $10^6$ dimensional multivariate random variable. What is the best (and fastest) way to do this? I am currently taking cholesky decomposition of the ...
0
votes
0answers
19 views

multi linear regression model. Modeling with a heavily skewed binary independent variable

Dataset and goal: One continuous measurement( to be modeled as a dependent variable) and four other measurements (one binary and the rest are category variables with multiple levels) to be modeled as ...
0
votes
0answers
14 views

What are the various ways to group a big time series (>2000) into few categories and apply one algorithm for each of these groups?

I have ~2000 time series to forecast. Do I need to be able to group them (for example, into 10) or find associations/dependencies between them (tests such as HHG, dCov etc.) so that I can use my ...
0
votes
0answers
34 views

How to compare groups when the control group is 100 times larger than the group in question?

I am working with NIS data which are basically a 20% sample of all in-hospital admissions in the US every year. I have small group of cases that carry diagnosis "X" in their record and overall the ...
1
vote
0answers
46 views

Server Requirements for machine learning on 300 million row database [closed]

I am trying to build a logistic regression model using Revolution R 64 bit. My data is 300 million rows by 12 columns. The data is stored on AWS Redshift, but takes about 6 hrs to import and save as ...
1
vote
0answers
17 views

How to store a large graph? [closed]

When I try to develop a spelling collector, I realize if I can include the word background (etc:before and after word) into the bayes calculation, may be could increase this spelling collector ...
0
votes
0answers
19 views

Help in understanding hashing for nearest neighbor search

Hashing is a technique for large- scale visual search and a variety of hashing-based method- s have been proposed Survey paper : Hashing for similarity search . The application of hashing to ...
0
votes
0answers
18 views

Hyperparameter optimization on large dataset

I have a huge dataset and want to carry out regression, such as gradient boosting. The problem is that the dataset is huge and hyperparameter optimization is computational expensive, especially I use ...
0
votes
0answers
33 views

How to visualize simplified dendrograms of many samples in R

I want to make sense out of my samples by visualizing my hierarchical clustering of a large dataset into a dendrogram in R. However as I have over 450k samples, I cannot make any sense out of the ...
0
votes
0answers
17 views

R model developing & validating - Open to Discussion [duplicate]

Throughout my R journey I have noticed the way we can use given data to develop and validate a model. Assume that you have given data for a problem train.csv test.csv Method A Combine ...
2
votes
0answers
18 views

Aggregating covariances

Suppose I have 2 data sets too large to combine. Each data set has one row per user, $i$, and two columns $(x, y)$. Users can be in both data sets but have different column values, $(x_{i1}, ...
3
votes
1answer
58 views

Why does a large sample size cause a significant ANOVA F-test?

If I have a large sample size, e.g. 100,000 data points, I know that most significance tests are going to come back with a very small p-value unless the null hypothesis is "true on the nose." In other ...
1
vote
2answers
85 views

Cross Validation - purpose, need and utility [duplicate]

The question might sound like an old one but I haven't got satisfactory answers for a number of questions I have about CV. I looked at several questions on CV here, here, here and here and yet things ...
2
votes
0answers
36 views

how to model longitudinal big data?

Traditionally we use mixed model to model longitudinal data, i.e. data like: ...
0
votes
1answer
35 views

How to plot a line graph for very huge data file? [closed]

How to plot a graph for very huge data. I am data sets containing minimum of 300000 counts. I need to plot a graph. Is there any special package for it. Also, I need to insert secondary axis for the ...
0
votes
0answers
17 views

Which statistical assumptions are still important when fitting a GLM to > 1 million observations?

I have previously only fit GLM models to small/medium sized data (up to several thousand points, maybe tens of thousands). I always try to be meticulous about checking that GLM assumptions hold where ...
0
votes
0answers
14 views

Database and Data Waehouse design and processing in Big Data

With the growing popularity of Big Data and related tools, I am looking for books that specifically deal with implementing a warehouse in one of Big Data technologies, database and data-warehouse ...
0
votes
1answer
24 views

Binary random variable, big data frame: does my approach make sense?

I have a large data frame with about 1100 columns containing integers and about 30'000 rows. The last column contains a binary random variable which attains values 0 and 1. 30% of the data frame ...
3
votes
0answers
61 views

How to do dimensionality reduction on a huge data set?

I am working with fMRI data of ~1000 subject. Each subject has a feature vector of ~150 million dimension. So I can only keep the feature vectors of ~10 subjects in memory. What are some algorithms ...
0
votes
0answers
23 views

Using Mann-Whitney U instead of T-test in large sample

For my dissertation, I am examining racial differences in perceived choice and control and home and community-based services (HCBS) use in frail elders (n=659) who have moved from a nursing home into ...
0
votes
0answers
21 views

Best way to read HDFS data in HBase

I have a source where real time data is streaming into Flume and then Flume dumps it into HDFS. QUESTION: I believe that there are couple of ways to read HDFS file in HBase. Can anyone brief me about ...
0
votes
0answers
19 views

Evaluate fit of classification model in large sample

I have a large data sample with around n=300,000 observations. My dependent variable is categorical (0: no, 1: yes) and I have 10 independent variables which are categorized into 2 or 3. I run ...
0
votes
0answers
29 views

Time series pattern match in large datasets

I want to find in a time series pattern that look similar to a predifined one. At the moment I am using DTW to do this and it works well. But now I want to use it for larger datasets like finding a ...
1
vote
0answers
33 views

Cluster analysis of large, multiple answer dataset

I have to analyse data from a marketing study. I will use SPSS. The questionnaire will look like this: Q: Imagine Situation X. Select 1-3 Criteria from the list that best describe your feeling. ...
1
vote
1answer
61 views

SVM classifier - can I average multiple models?

I'm performing SVM classification on a relatively large data set (~1M rows, 4 variables). I want to assign a classification score to each row, not evaluate input parameters, so following the top ...
1
vote
0answers
49 views

Statistical testing in machine learning [closed]

I have a clarification. I am currently working as a data scientist for an IT company. I have experience in the area of NLP, ML and Big data for more than 5 years. I am from Bioinformatics background. ...
2
votes
0answers
34 views

What systems are based on the following paper? [closed]

http://papers.nips.cc/paper/3150-map-reduce-for-machine-learning-on-multicore.pdf This paper has Andrew Ng as one of the authors. I really liked the idea presented in this paper, and when I looked ...
1
vote
1answer
165 views

K-Means Clustering using modified correlation (1 - pearson correlation coefficient)

I am trying to implement k-means clustering on a 6x6 data set that looks like this: 2 3 6 0 1 7 4 9 9 6 2 2 0 1 7 9 5 0 2 3 2 7 8 3 8 2 9 2 3 1 8 0 0 1 7 9 Using ...
17
votes
1answer
371 views

Do we really perform multivariate regression analysis with *million* coefficients/independent variables?

I'm spending some time learning machine learning (sorry for the recursion :) and I couldn't help be intrigued by the rule of thumb of choosing Gradient Descent over direct equation solving for ...
1
vote
1answer
38 views

Can increasing the significance level help to avoid detecting minor differences in large datasets?

In the accepted answer of THIS question, it is stated that "With such large sample sizes both tests will have high power to detect minor differences". On the other hand, a voted up comment in THIS ...
0
votes
0answers
14 views

Ignore data where the minutes modulo five are not equal to zero

I've some data that I'll analyse. Every five minutes there was taken the power (P) of solar panels over five years. I've calculate the kilo watt hour of that moment by this formule: $$P[Wh] = P[W] ⋅ ...
0
votes
0answers
16 views

Performance issue doing FAMD(Factor Analysis for Mixed Data)

not sure if it goes here or on SO, tell me. I code in R. I have a huge dataset, ~150 variables and 250k rows, ~20 qualitative, 130 quantitative. I want to perform a dimension reduction to give me an ...
1
vote
1answer
47 views

Confused about the sample size in large sample confidential interval and population proportion intertal

My professor said when sample size is large, it will has the following formula, but how big is the sample can be considered to be a large sample and use the following formula? I did some questions ...
2
votes
1answer
53 views

Intuition with chi squared test and large frequencies

I'm working with categorical data and want to run a chi squared test. I want to test the frequencies: A = list(12455, 11554, 11908, 13416, 12647, 7828, 10172) ...
2
votes
2answers
49 views

How confident can I be that a proportion I found in a sample is true of the population?

Out of a population of roughly 2.0 million I took a random sample using the following parameters : Confidence Level: 95% Confidence Interval: 5 The sample I got (384 items as recommended by ...
3
votes
2answers
95 views

Mann-Whitney (U) Test at N>20

I am a high school Psychology student. I just performed a study where two separate groups of sizes 26 and 28 took the same test under different conditions. For the data analysis my professor has ...
-1
votes
1answer
325 views

Weka - Run K-Means++ Algorithm in JAVA code to preserve memory

Anyone know how to run weka k-means++ clustering source directly in JAVA code to preserve memory? I load and run k-means++ clustering for large datasets (6 millions) in weka but always freeze, i try ...
0
votes
0answers
44 views

Distance-based sequential clustering for large data

Let's say I have a sequence of data point $x_1,\dots, x_n\in \Bbb R^d$. I would like to build clusters as follows: I give some $\varepsilon$, and start with $x_1$. At each step $k$ I add $x_{k+1}$ to ...
0
votes
0answers
35 views

Maximum number of iterations (in function “homals” in R) and transformation nonlinear data to linear data

is there anyone who knows what does "maximal number of iterations" can mean in nonlinear PCA (in "homals" function in R) and how to transform nonlinear data to linear data (in R software or in ...
0
votes
0answers
28 views

Prediction of Probability of being Late

I wonder if there is any insight or book recommendation on the calculation of Late Index. Late Index should be a number that are suggestive on the probability of an individual being late or not. Says ...