'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

0
votes
0answers
16 views

multi linear regression model. Modeling with a heavily skewed binary independent variable

Dataset and goal: One continuous measurement( to be modeled as a dependent variable) and four other measurements (one binary and the rest are category variables with multiple levels) to be modeled as ...
0
votes
0answers
12 views

What are the various ways to group a big time series (>2000) into few categories and apply one algorithm for each of these groups?

I have ~2000 time series to forecast. Do I need to be able to group them (for example, into 10) or find associations/dependencies between them (tests such as HHG, dCov etc.) so that I can use my ...
0
votes
0answers
27 views

How to compare groups when the control group is 100 times larger than the group in question?

I am working with NIS data which are basically a 20% sample of all in-hospital admissions in the US every year. I have small group of cases that carry diagnosis "X" in their record and overall the ...
1
vote
0answers
38 views

Server Requirements for machine learning on 300 million row database [closed]

I am trying to build a logistic regression model using Revolution R 64 bit. My data is 300 million rows by 12 columns. The data is stored on AWS Redshift, but takes about 6 hrs to import and save as ...
1
vote
0answers
15 views

How to store a large graph? [closed]

When I try to develop a spelling collector, I realize if I can include the word background (etc:before and after word) into the bayes calculation, may be could increase this spelling collector ...
0
votes
0answers
16 views

Help in understanding hashing for nearest neighbor search

Hashing is a technique for large- scale visual search and a variety of hashing-based method- s have been proposed Survey paper : Hashing for similarity search . The application of hashing to ...
0
votes
0answers
14 views

Hyperparameter optimization on large dataset

I have a huge dataset and want to carry out regression, such as gradient boosting. The problem is that the dataset is huge and hyperparameter optimization is computational expensive, especially I use ...
0
votes
0answers
16 views

How to visualize simplified dendrograms of many samples in R

I want to make sense out of my samples by visualizing my hierarchical clustering of a large dataset into a dendrogram in R. However as I have over 450k samples, I cannot make any sense out of the ...
0
votes
0answers
17 views

R model developing & validating - Open to Discussion [duplicate]

Throughout my R journey I have noticed the way we can use given data to develop and validate a model. Assume that you have given data for a problem train.csv test.csv Method A Combine ...
2
votes
0answers
15 views

Aggregating covariances

Suppose I have 2 data sets too large to combine. Each data set has one row per user, $i$, and two columns $(x, y)$. Users can be in both data sets but have different column values, $(x_{i1}, ...
0
votes
0answers
28 views

Clustering variables based on similar trends

EDIT: Due to lack of any answers, I'll try to give more detail and expand the question with more information. I have a large dataset (millions of variables, tens of columns) which represents ...
3
votes
1answer
52 views

Why does a large sample size cause a significant ANOVA F-test?

If I have a large sample size, e.g. 100,000 data points, I know that most significance tests are going to come back with a very small p-value unless the null hypothesis is "true on the nose." In other ...
1
vote
2answers
63 views

Cross Validation - purpose, need and utility [duplicate]

The question might sound like an old one but I haven't got satisfactory answers for a number of questions I have about CV. I looked at several questions on CV here, here, here and here and yet things ...
2
votes
0answers
26 views

how to model longitudinal big data?

Traditionally we use mixed model to model longitudinal data, i.e. data like: ...
0
votes
1answer
30 views

How to plot a line graph for very huge data file? [closed]

How to plot a graph for very huge data. I am data sets containing minimum of 300000 counts. I need to plot a graph. Is there any special package for it. Also, I need to insert secondary axis for the ...
0
votes
0answers
14 views

Which statistical assumptions are still important when fitting a GLM to > 1 million observations?

I have previously only fit GLM models to small/medium sized data (up to several thousand points, maybe tens of thousands). I always try to be meticulous about checking that GLM assumptions hold where ...
0
votes
0answers
14 views

Database and Data Waehouse design and processing in Big Data

With the growing popularity of Big Data and related tools, I am looking for books that specifically deal with implementing a warehouse in one of Big Data technologies, database and data-warehouse ...
0
votes
1answer
22 views

Binary random variable, big data frame: does my approach make sense?

I have a large data frame with about 1100 columns containing integers and about 30'000 rows. The last column contains a binary random variable which attains values 0 and 1. 30% of the data frame ...
3
votes
0answers
55 views

How to do dimensionality reduction on a huge data set?

I am working with fMRI data of ~1000 subject. Each subject has a feature vector of ~150 million dimension. So I can only keep the feature vectors of ~10 subjects in memory. What are some algorithms ...
0
votes
0answers
20 views

Using Mann-Whitney U instead of T-test in large sample

For my dissertation, I am examining racial differences in perceived choice and control and home and community-based services (HCBS) use in frail elders (n=659) who have moved from a nursing home into ...
0
votes
0answers
18 views

Best way to read HDFS data in HBase

I have a source where real time data is streaming into Flume and then Flume dumps it into HDFS. QUESTION: I believe that there are couple of ways to read HDFS file in HBase. Can anyone brief me about ...
0
votes
0answers
16 views

Evaluate fit of classification model in large sample

I have a large data sample with around n=300,000 observations. My dependent variable is categorical (0: no, 1: yes) and I have 10 independent variables which are categorized into 2 or 3. I run ...
0
votes
0answers
26 views

Time series pattern match in large datasets

I want to find in a time series pattern that look similar to a predifined one. At the moment I am using DTW to do this and it works well. But now I want to use it for larger datasets like finding a ...
1
vote
0answers
30 views

Cluster analysis of large, multiple answer dataset

I have to analyse data from a marketing study. I will use SPSS. The questionnaire will look like this: Q: Imagine Situation X. Select 1-3 Criteria from the list that best describe your feeling. ...
1
vote
1answer
53 views

SVM classifier - can I average multiple models?

I'm performing SVM classification on a relatively large data set (~1M rows, 4 variables). I want to assign a classification score to each row, not evaluate input parameters, so following the top ...
1
vote
0answers
45 views

Statistical testing in machine learning [closed]

I have a clarification. I am currently working as a data scientist for an IT company. I have experience in the area of NLP, ML and Big data for more than 5 years. I am from Bioinformatics background. ...
2
votes
0answers
32 views

What systems are based on the following paper? [closed]

http://papers.nips.cc/paper/3150-map-reduce-for-machine-learning-on-multicore.pdf This paper has Andrew Ng as one of the authors. I really liked the idea presented in this paper, and when I looked ...
1
vote
1answer
134 views

K-Means Clustering using modified correlation (1 - pearson correlation coefficient)

I am trying to implement k-means clustering on a 6x6 data set that looks like this: 2 3 6 0 1 7 4 9 9 6 2 2 0 1 7 9 5 0 2 3 2 7 8 3 8 2 9 2 3 1 8 0 0 1 7 9 Using ...
17
votes
1answer
357 views

Do we really perform multivariate regression analysis with *million* coefficients/independent variables?

I'm spending some time learning machine learning (sorry for the recursion :) and I couldn't help be intrigued by the rule of thumb of choosing Gradient Descent over direct equation solving for ...
1
vote
1answer
37 views

Can increasing the significance level help to avoid detecting minor differences in large datasets?

In the accepted answer of THIS question, it is stated that "With such large sample sizes both tests will have high power to detect minor differences". On the other hand, a voted up comment in THIS ...
0
votes
0answers
14 views

Ignore data where the minutes modulo five are not equal to zero

I've some data that I'll analyse. Every five minutes there was taken the power (P) of solar panels over five years. I've calculate the kilo watt hour of that moment by this formule: $$P[Wh] = P[W] ⋅ ...
0
votes
0answers
15 views

Performance issue doing FAMD(Factor Analysis for Mixed Data)

not sure if it goes here or on SO, tell me. I code in R. I have a huge dataset, ~150 variables and 250k rows, ~20 qualitative, 130 quantitative. I want to perform a dimension reduction to give me an ...
1
vote
1answer
47 views

Confused about the sample size in large sample confidential interval and population proportion intertal

My professor said when sample size is large, it will has the following formula, but how big is the sample can be considered to be a large sample and use the following formula? I did some questions ...
2
votes
1answer
49 views

Intuition with chi squared test and large frequencies

I'm working with categorical data and want to run a chi squared test. I want to test the frequencies: A = list(12455, 11554, 11908, 13416, 12647, 7828, 10172) ...
2
votes
2answers
49 views

How confident can I be that a proportion I found in a sample is true of the population?

Out of a population of roughly 2.0 million I took a random sample using the following parameters : Confidence Level: 95% Confidence Interval: 5 The sample I got (384 items as recommended by ...
3
votes
2answers
87 views

Mann-Whitney (U) Test at N>20

I am a high school Psychology student. I just performed a study where two separate groups of sizes 26 and 28 took the same test under different conditions. For the data analysis my professor has ...
-1
votes
1answer
288 views

Weka - Run K-Means++ Algorithm in JAVA code to preserve memory

Anyone know how to run weka k-means++ clustering source directly in JAVA code to preserve memory? I load and run k-means++ clustering for large datasets (6 millions) in weka but always freeze, i try ...
0
votes
0answers
40 views

Distance-based sequential clustering for large data

Let's say I have a sequence of data point $x_1,\dots, x_n\in \Bbb R^d$. I would like to build clusters as follows: I give some $\varepsilon$, and start with $x_1$. At each step $k$ I add $x_{k+1}$ to ...
0
votes
0answers
33 views

Maximum number of iterations (in function “homals” in R) and transformation nonlinear data to linear data

is there anyone who knows what does "maximal number of iterations" can mean in nonlinear PCA (in "homals" function in R) and how to transform nonlinear data to linear data (in R software or in ...
0
votes
0answers
28 views

Prediction of Probability of being Late

I wonder if there is any insight or book recommendation on the calculation of Late Index. Late Index should be a number that are suggestive on the probability of an individual being late or not. Says ...
0
votes
1answer
23 views

Avoiding cluster recalculation in large scale clustering

I am working with a large dataset containing about 500 million records and about 50 discrete dimensions. I am planning to cluster these records into a number of distinct clusters (~30). Now, every ...
0
votes
0answers
34 views

Mortality time series forecasting

I am doing an analysis to estimate and forecast TB mortality in Bangladesh. My data has Date of death, Age, Sex, Marital status, Occupation. Data were collected from 2004 to 2009. It is a ...
2
votes
1answer
67 views

Distance metric invariant to dimensionality?

I'm working on a classification/prediction problem where I have to predict a location of an object. The problem that I have is that for every location, I have a unique and different number of feature ...
41
votes
9answers
7k views

What exactly is Big Data?

I have been asked on several occasions the question: What is Big-Data? Both by students and my relatives that are picking up the buzz around statistics and ML. I found this CV-post. And I feel ...
1
vote
1answer
189 views

Collinearity in R for dataset with 40+ variables?

I have a big data matrix with 6000 rows (observations) and 45 columns (44 predictive variables (categorical and continuous) and 1 response variable (0 or 1). I want to check the correlation/ ...
0
votes
1answer
92 views

Classifier and Technique to use for large number of categories

I am designing a scikit learn classifier which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have already tried with all the ...
0
votes
1answer
318 views

R - Classification ctree {party} - Testing sample and leaf attribution with unbalanced data

Let's start with data description of the website visits I analyse : 6M rows Dependant variable quotation is binary and takes values ...
2
votes
1answer
111 views

Estimation of quantile given quantiles of subset

Let's say we have sets $X=\{x_1, x_2, \ldots, x_m\}, Y=\{y_1, y_2, \ldots, y_n\}$ and we have some estimate (or exact) quantile information about them at some level $a$. How could we approximate the ...
0
votes
1answer
97 views

Is the Cohen's D a suitable test for my dataset?

I have two variables (A and B). A has 3,000 samples; B ...