'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

learn more… | top users | synonyms (2)

0
votes
0answers
12 views

Using Mann-Whitney U instead of T-test in large sample

For my dissertation, I am examining racial differences in perceived choice and control and home and community-based services (HCBS) use in frail elders (n=659) who have moved from a nursing home into ...
0
votes
0answers
7 views

Best way to read HDFS data in HBase

I have a source where real time data is streaming into Flume and then Flume dumps it into HDFS. QUESTION: I believe that there are couple of ways to read HDFS file in HBase. Can anyone brief me about ...
0
votes
0answers
16 views

Evaluate fit of classification model in large sample

I have a large data sample with around n=300,000 observations. My dependent variable is categorical (0: no, 1: yes) and I have 10 independent variables which are categorized into 2 or 3. I run ...
0
votes
0answers
17 views

Time series pattern match in large datasets

I want to find in a time series pattern that look similar to a predifined one. At the moment I am using DTW to do this and it works well. But now I want to use it for larger datasets like finding a ...
1
vote
0answers
21 views

Cluster analysis of large, multiple answer dataset

I have to analyse data from a marketing study. I will use SPSS. The questionnaire will look like this: Q: Imagine Situation X. Select 1-3 Criteria from the list that best describe your feeling. ...
1
vote
1answer
32 views

SVM classifier - can I average multiple models?

I'm performing SVM classification on a relatively large data set (~1M rows, 4 variables). I want to assign a classification score to each row, not evaluate input parameters, so following the top ...
1
vote
0answers
36 views

Statistical testing in machine learning [closed]

I have a clarification. I am currently working as a data scientist for an IT company. I have experience in the area of NLP, ML and Big data for more than 5 years. I am from Bioinformatics background. ...
2
votes
0answers
32 views

What systems are based on the following paper? [closed]

http://papers.nips.cc/paper/3150-map-reduce-for-machine-learning-on-multicore.pdf This paper has Andrew Ng as one of the authors. I really liked the idea presented in this paper, and when I looked ...
1
vote
1answer
75 views

K-Means Clustering using modified correlation (1 - pearson correlation coefficient)

I am trying to implement k-means clustering on a 6x6 data set that looks like this: 2 3 6 0 1 7 4 9 9 6 2 2 0 1 7 9 5 0 2 3 2 7 8 3 8 2 9 2 3 1 8 0 0 1 7 9 Using ...
16
votes
1answer
334 views

Do we really perform multivariate regression analysis with *million* coefficients/independent variables?

I'm spending some time learning machine learning (sorry for the recursion :) and I couldn't help be intrigued by the rule of thumb of choosing Gradient Descent over direct equation solving for ...
1
vote
1answer
34 views

Can increasing the significance level help to avoid detecting minor differences in large datasets?

In the accepted answer of THIS question, it is stated that "With such large sample sizes both tests will have high power to detect minor differences". On the other hand, a voted up comment in THIS ...
0
votes
0answers
12 views

Ignore data where the minutes modulo five are not equal to zero

I've some data that I'll analyse. Every five minutes there was taken the power (P) of solar panels over five years. I've calculate the kilo watt hour of that moment by this formule: $$P[Wh] = P[W] ⋅ ...
0
votes
0answers
10 views

Performance issue doing FAMD(Factor Analysis for Mixed Data)

not sure if it goes here or on SO, tell me. I code in R. I have a huge dataset, ~150 variables and 250k rows, ~20 qualitative, 130 quantitative. I want to perform a dimension reduction to give me an ...
1
vote
1answer
33 views

Confused about the sample size in large sample confidential interval and population proportion intertal

My professor said when sample size is large, it will has the following formula, but how big is the sample can be considered to be a large sample and use the following formula? I did some questions ...
2
votes
1answer
41 views

Intuition with chi squared test and large frequencies

I'm working with categorical data and want to run a chi squared test. I want to test the frequencies: A = list(12455, 11554, 11908, 13416, 12647, 7828, 10172) ...
2
votes
2answers
43 views

How confident can I be that a proportion I found in a sample is true of the population?

Out of a population of roughly 2.0 million I took a random sample using the following parameters : Confidence Level: 95% Confidence Interval: 5 The sample I got (384 items as recommended by ...
3
votes
2answers
66 views

Mann-Whitney (U) Test at N>20

I am a high school Psychology student. I just performed a study where two separate groups of sizes 26 and 28 took the same test under different conditions. For the data analysis my professor has ...
-1
votes
1answer
177 views

Weka - Run K-Means++ Algorithm in JAVA code to preserve memory

Anyone know how to run weka k-means++ clustering source directly in JAVA code to preserve memory? I load and run k-means++ clustering for large datasets (6 millions) in weka but always freeze, i try ...
0
votes
0answers
31 views

Distance-based sequential clustering for large data

Let's say I have a sequence of data point $x_1,\dots, x_n\in \Bbb R^d$. I would like to build clusters as follows: I give some $\varepsilon$, and start with $x_1$. At each step $k$ I add $x_{k+1}$ to ...
0
votes
0answers
31 views

Maximum number of iterations (in function “homals” in R) and transformation nonlinear data to linear data

is there anyone who knows what does "maximal number of iterations" can mean in nonlinear PCA (in "homals" function in R) and how to transform nonlinear data to linear data (in R software or in ...
0
votes
0answers
26 views

Prediction of Probability of being Late

I wonder if there is any insight or book recommendation on the calculation of Late Index. Late Index should be a number that are suggestive on the probability of an individual being late or not. Says ...
0
votes
1answer
17 views

Avoiding cluster recalculation in large scale clustering

I am working with a large dataset containing about 500 million records and about 50 discrete dimensions. I am planning to cluster these records into a number of distinct clusters (~30). Now, every ...
0
votes
0answers
31 views

Mortality time series forecasting

I am doing an analysis to estimate and forecast TB mortality in Bangladesh. My data has Date of death, Age, Sex, Marital status, Occupation. Data were collected from 2004 to 2009. It is a ...
2
votes
1answer
60 views

Distance metric invariant to dimensionality?

I'm working on a classification/prediction problem where I have to predict a location of an object. The problem that I have is that for every location, I have a unique and different number of feature ...
41
votes
9answers
6k views

What exactly is Big Data?

I have been asked on several occasions the question: What is Big-Data? Both by students and my relatives that are picking up the buzz around statistics and ML. I found this CV-post. And I feel ...
1
vote
1answer
119 views

Collinearity in R for dataset with 40+ variables?

I have a big data matrix with 6000 rows (observations) and 45 columns (44 predictive variables (categorical and continuous) and 1 response variable (0 or 1). I want to check the correlation/ ...
0
votes
1answer
77 views

Classifier and Technique to use for large number of categories

I am designing a scikit learn classifier which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have already tried with all the ...
0
votes
1answer
180 views

R - Classification ctree {party} - Testing sample and leaf attribution with unbalanced data

Let's start with data description of the website visits I analyse : 6M rows Dependant variable quotation is binary and takes values ...
2
votes
1answer
97 views

Estimation of quantile given quantiles of subset

Let's say we have sets $X=\{x_1, x_2, \ldots, x_m\}, Y=\{y_1, y_2, \ldots, y_n\}$ and we have some estimate (or exact) quantile information about them at some level $a$. How could we approximate the ...
0
votes
1answer
81 views

Is the Cohen's D a suitable test for my dataset?

I have two variables (A and B). A has 3,000 samples; B ...
2
votes
0answers
78 views

Structuring Many-Factor Data for Linear Regression in R

I have a fairly large dataset of the following form, and I want to run a linear regression returning coefficients for each factor: ...
0
votes
0answers
11 views

Ranking of significant (and insignificant) coefficients of regional dummies

I have multiple regression of dependent variable on some regional dummies. I see in a paper (Oswald 2010) that the author rank estimated coefficients of the state dummies. My sample size is about ...
0
votes
0answers
52 views

Core vector machine implementation

I came across the following article : http://www.jmlr.org/papers/volume6/tsang05a/tsang05a.pdf, Core Vector Machines: Fast SVM Training on Very Large Data Sets. The approach looks very promising, ...
2
votes
2answers
300 views

Fisher's test with large data

I want to compare the proportions in two samples, that can be organized as P NP SAMPLE_A 3,129,548 427,953 SAMPLE_B 2,930,639 407,353 If I ...
2
votes
0answers
67 views

Survival analysis with Frailty on large dataset

I am trying to fit a survival analysis in R with non-recurrent events and time-varying coefficients. The baseline distribution is exponential or Weibull and the ...
1
vote
1answer
43 views

How to apply distance-based clustering or dimensionality reduction for too many samples

I have a dataset with 200K samples (cases) and 30 variables. Every distance-based method for clustering or dimension reduction technique that I use, such as DBSCAN, Hierarchical Clustering, LLE, ...
0
votes
0answers
45 views

Training set selection

I have the following question for a project I'm working on. I am trying to find the best strategy to select the best training set in a dataset. I have a dataset with a few billions rows. I am trying ...
1
vote
0answers
45 views

Sequential conditional simulation to avoid using a large covariance matrix

I would like to generate $S$ samples of a $T \cdot M$ dimensional vector, where $T$ is the number of time steps and $M$ the number of locations, i.e., the vector is a stack with $T$ values for ...
1
vote
1answer
102 views

Which statistical analysis to use to compare the level of similarity between two large samples?

I'm writing a small speech recognition prototype as my side project, which matches pre-recorded words of the speaker. So now I'm thinking of comparing two sets of data (outcome of FFT) which are two ...
0
votes
2answers
72 views

R: Multinomial Logistic Regression for health data

I am doing some data analysis on a fairly large health data set of patients with diagnoses and the respective procedures received for each event. I was asked to run a multinomial logistic regression ...
1
vote
1answer
39 views

Pattern Recognition - Visualizing Dense Data Points

I have a sample of around 5000 data (2D) points that are generated through a simulation of a cryptocurrency's mining events of following form. In column 2 one can see identical y-values with ...
0
votes
2answers
186 views

Cluster Analysis for large data in R

I am trying to perform a clustering analysis for a csv file with 50k+ rows, 10 columns. I tried k-mean, hierarchical and model based clustering methods. Only k-mean works because of the large data ...
1
vote
0answers
27 views

Taming large datasets. How to keep historical data with aggregation techniques?

I will be collecting huge amounts of computer performance data and the datasets will get large very quickly. This necessitates reducing the size of the data by compressing the historical data using ...
1
vote
2answers
121 views

Silhouette clustering index in practice

I don't have much experience with data analysis algorithms (data mining, machine learning, if you like) and I'm interested if some could share their experience with practical usage of Silhouette in ...
0
votes
0answers
34 views

Multivariate discretization method / library for huge data

Does anyone know any multivariate discretization method that can be used for large amounts of data. A library / Python library would be awesome but algorithms would also do. Also I'm not sure if ...
0
votes
0answers
47 views

What is the best way to test normality for a very large sample size 15000? [duplicate]

Also do I need to test the normality for the categorical data? Thanks
2
votes
0answers
69 views

Classification and regression tree (CART) on large data set

I am trying to approximate a multivariate function $y = f(x_1, ...x_n)$, which I have reason to believe will be well approximated by a classification and regression tree. Some of the variables are ...
0
votes
2answers
48 views

Increasing the power by dropping points: can I do it?

I am repeating a test on a large amount of data and FDR-correcting the p-values afterwards for multiple testing. Yet I still do not have enough power. However, I feel like it is not necessary to test ...
0
votes
0answers
64 views

Visualize multivariate data in Excel

Here is an example of the data I want to visualize either as stacked bar or as scatter plot. ...
1
vote
1answer
103 views

Machine learning tutorials / examples on data sets larger than a terabyte

I am trying to gather a list of practical ML examples / tutorials on more than a terabyte of data. I'm particularly interested in feature extraction from large data sets that involves aggregation (the ...