Questions tagged [large-data]

'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

Filter by
Sorted by
Tagged with
5
votes
2answers
90 views

Huge sample sizes, tests, and deviation of assumptions?

I am performing Wilcoxon test (but in theory it can be any) and sometimes my sample sizes are huge. Even a single outlier may cause an extremely low p value. I am not interested in small effects at ...
0
votes
0answers
26 views

Data analysis in large samples

I am a graduate student in Mathematics and I am supposed to conduct a seminar concerning problems in data analysis, especially regarding large samples. I am looking for some resources providing ...
1
vote
1answer
32 views

What test to test difference between two independent groups of (very) unequal size?

My goal is to test the difference between two independent and non parametric samples of unequal size (3191 and 2209). One sample concerns questions and the other one responses and the data concerns ...
0
votes
0answers
9 views

Interpretation of the Hausman test (overidentification in relation to IV's) for large samples

I am using survey data with a huge amount of observations, such as the World Value Surveys. Large sample sizes are obviously very nice, but I have have encountered some downsides as well. To give an ...
3
votes
2answers
65 views

Learning from multiple very varied data sets?

Suppose we have a set of objects $X$ (e.g. individual humans). Suppose also that humans can be described by a set of (potentially very high-dimensional) variables $V_i$, (e.g. $V_1$ is a picture of ...
0
votes
0answers
10 views

Require suggestions on data handling

I have below tables in my database with record count as shown below I would like to join all these tables and make one final/main table. I tried the below approaches but nothing helped 1) Currently ...
0
votes
0answers
11 views

Question About Coming Up With Own Function for Distance Matrix (For Clustering)

Right now, I am currently working on implementing a clustering algorithm with millions data entries with regards to game users for a mobile game. A lot of the features I plan on using are unique to ...
3
votes
1answer
163 views

Difference between big data and high dimensional data

What do big data and high dimensional data mean? Is high dimensional data a special case of big data? What are the complications that arise in the analysis of high dimensional and big data each?
0
votes
0answers
14 views

Tests if k-samples are from same population are rejected for large N?

For a very large data set I have created a number of samples and try to verify whether the distribution of the variables in those samples is similar. I.e. I'm trying to verify that the sampling ...
3
votes
1answer
42 views

Statistical Significance in Table Ones

I have a question about statistical and clinical significance. In a non-clinical trials setting (e.g. observational), if the sample size is large for both groups (assume 2 groups for simplicity) ...
0
votes
1answer
76 views

Suggestions to cluster more than 300k observations

I am trying to perform an hierarchical clustering on data frame that contains 300k records 7 features (3 binaries and 4 continuous) in order to get insights on what looks like my dataset. I've chosen ...
0
votes
1answer
26 views

Detrending for a large dataset consisting of many group time-series

I have a huge dataset consisting of many individuals (~20000), each with a month of daily data. I am thinking of detrending those individual time series that are non-stationary but visually ...
0
votes
1answer
80 views

Multivariate and high-dimensional data, are they the same?

I read about the multivariate and high-dimensional data set. I found that the multivariate data is the data with more than 3 variables. In addition, the high-dimensional data is the data with a large ...
0
votes
0answers
49 views

Standard deviation for proportion with huge denominator

My question involves finding the standard deviation of a ratio when the denominator is huge. The standard formula $$\sqrt{\frac{p(1−p)}{n}}$$ yields a number that is almost zero. The context is ...
2
votes
1answer
134 views

When to prefer PCA over regularization methods in regression?

When dealing with the curse of dimensionality, regularization methods seem to be clear in their intuition. All "regularization" methods can be seen as a "squeezing" of one's variables towards ...
0
votes
1answer
33 views

Estimating probability density function of big amount of data coming from MC simulations

I am trying to estimate Probability Density Function (PDF) of a big amount of data ($1e^6$ , $1e^7$, and higher) coming from Mote Carlo (MC) simulation. My objective is to estimate the PDF (e.g. with ...
1
vote
0answers
26 views

Summary statistics for bipartite networks

I have a large bipartite network that I would like to summarise. So far, I have found the following summary statistics: Degree centrality Graph density Modularity Nestedness I have not found a ...
0
votes
1answer
20 views

Large scale SVM classification problem

Problem I am now working on a sentiment analysis task where the largest dataset involves 36 million custom reviews and associated sentiment (positive or negative). The feature extraction process is ...
1
vote
0answers
147 views

How to do exploratory analysis on very large dataset?

I have very large dataset stored in Google Cloud BigQuery. (the system dumps a table everyday. A table has approximately 25GB of size, and I have 2 months data = 60 tables = 60x25 GB) It is quite ...
0
votes
0answers
13 views

Difference between retraining on different portions of data and training initially on larger data set

I have a large data set that doesn't fit in memory and would have to use something like Keras's model.fit_generator if I would like to train the model on all of the ...
0
votes
0answers
42 views

Chi-Square Degeneracy for Large Sample

(Forgive my hand-waving explanation) When discussing anomaly detection methods (for example), one possibility is comparing the distance of a point from a centroid: Given 100 samples $X_1,...,X_{100}$ ...
0
votes
0answers
15 views

Compare and visualization of multiple output data files

I have 100 data files corresponding to 100 samples. For these 100 data files, I have processed data in 4 different ways to generate result output files. Out of the 4 results files per sample, I ...
0
votes
1answer
155 views

How to handle or impute large number of missing values?

I am trying to use this dataset to build a predictive model. The hubway.db file contains 3 tables. One of which is is bike_trips...
0
votes
2answers
81 views

Sampling Big Data for Machine Learning [closed]

In practice, how does one go about sampling a from big data set (eg. +/- 50 million distinct observations) to perform ML using Python? Most non-parametric models (e.g., SVM, ensemble models) start to ...
3
votes
0answers
52 views

How can statistics be used to avoid “Lending False Credibility To Decisions We've Already Made”

In light of this article Data Science Has Become About Lending False Credibility To Decisions We've Already Made published in Forbes, I would appreciate input from the statistical and data science ...
0
votes
1answer
40 views

Multiple Entries for Same Participant [closed]

I have raw data that I need to transform and unsure as to how. Manually doing it is out of the question due to the thousands of entries. I have the data in excel and I’m looking to analyze in SPSS. ...
1
vote
1answer
134 views

Fitting a multivariate Gaussian with extremely sparse samples

We have a multi-variate Gaussian distribution. For instance with 3 variables. The correlations between the variables are important! We are fitting it to data, however, the samples are such that each ...
1
vote
2answers
50 views

General question: How do you visualize/deal with a lot of predictors?

In STATS classes, one is always taught to draw a picture to look for outliers, to look for the distribution type, to look for patterns in general. However, when you have a dataset with a lot of ...
0
votes
0answers
43 views

Feature Selection with interactions in high dimensions

Is there any fast approach to find features considering interactions in many variables (~3000)? Many methods like RFE applying random forest would take very long. I tried MARS with degree=2 but it ...
0
votes
0answers
26 views

Clarification on quantification of Categorical variables

I have a countries column with 49 levels. I want to quantify it. If I run CATPCA on that column would i be able to get the quantified result. Since CatPCA is like PCA or factor analysis: it extracts ...
1
vote
1answer
58 views

Machine learning model underperformance on unseen data

This is a follow-up question to a question I had previously posted on this forum We conducted an experiment on 100 subjects and obtained a dataset that was used to train a machine learning model that ...
0
votes
0answers
13 views

Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
0
votes
0answers
12 views

Large datasets, deviations and noise/signal

The below excerpt mentions that 'large deviations' are more attributable to variance than to information. What does the author mean when he says 'deviations'? Deviations from what? And why would it be ...
0
votes
0answers
34 views

Comparing a distributions between large datasets

I have $2 < n < 10$ algorithms that I have applied to $10 < n < 500$ time series with ten different variables. The job of the algorithms is to produce a result that is as close as possible ...
6
votes
4answers
497 views

Does sampling from a large dataset lead to correct inferences?

Say we have some population, and we obtain a "representative" random sample of that population, $(y_i, x_i)_{i = 1}^n$, where $n$ is very large (millions) and $x_i = (x_{i1}, x_{i2}, ... x_{ip})'$ is ...
0
votes
1answer
40 views

How to interpret a given 2D co-variance matrix?

I am trying to solve a problem regarding revision for my Big Data module. I have two main questions. 1) Given a predefined co-variance matrix: A cluster of points is distributed in a two-...
0
votes
1answer
79 views

Calculate standard error for very large number of observations [closed]

I have a very large dataset (with > 2 million simulated values). I want to compute standard error for this dataset. To do that, I divide the standard deviation by square root of number of observations....
0
votes
0answers
132 views

What are some fast outlier detection methods for big data in R?

I have a large dataset (300,000 rows) for which there are clear outliers. Box plots of two of the DVs of interest reveal the presence of large numbers of outliers by the Tukey outlier detection rule (...
1
vote
1answer
75 views

Duplicated Rows in Mixed Data Type Clustering

I have a dataset which has ~200k rows and looks like the following - ...
1
vote
1answer
32 views

How to perform specific queries in weather data time series [closed]

I have time series data from several weather stations located in a specific area. The readings include a timestamp, the humidity and the temperature. The resolution of the data is quite high, about 6 ...
0
votes
0answers
41 views

t-test advice for simulations having multiple runs [duplicate]

Here's the gist of the problem - I am simulating n scenarios, each resulting from changing various simulation parameters. I want to demonstrate that I can offer a mechanism that would be able to ...
0
votes
0answers
27 views

How important is research on model selection methods in Statistics?

My question is nothing technical. I just wanted your opinion on how important is the model selection problem in the field of Statistics considering the age of big data. Are the current methods such as ...
1
vote
2answers
111 views

Where can I find high-dimensional (p>n) datasets? [closed]

I am looking for "high-dimensional" data for a course project. The requirements of an ideal dataset for me are: 1.$p>n$ (or at least $p> \sqrt{n}$), where $p$ is the number of variables and $...
0
votes
0answers
47 views

residual plot for positive variable

I have a linear regression where the output variable is always greater equal zero with a nontrivial weight at exactly zero. The predictions of the model are also always greater than zero. What should ...
1
vote
1answer
214 views

How do I get the density of a region in a vector space?

I have a simple problem, which I think must have an easy solution. I have a vector space say with a 1000 dimensions for each vector. Now, I have a large number of sample vectors from this vector ...
0
votes
1answer
33 views

Deep learning with a lot of training data

I am building a bidirectional LSTM to do a sequential text-tagging task (particularly, automatic punctuation). Usually, the training is done in iterations, where in each iteration, the entire training ...
2
votes
0answers
298 views

Missing data imputation that can handle large data

I am looking for a reasonably scaling missing data imputation approach for big data (e.g. a well-scaling version of kNN - the standard versions we tried so far just ran out of memory) that fulfills ...
1
vote
0answers
75 views

How to combine multiple kernels of large sample datasets?

I have multiple large sample datasets in matrix format (each has 15000 rows and 5-50 columns) corresponding to different experiments. Each matrix contains the same number of samples(rows) but the ...
1
vote
1answer
46 views

From Big Data to a normal regression problem

My goal is to predict taxi demand depending on location and hour in NYC. I constructed a large dataset with ~19 million observations. However, it is computationally very expensive to perform ...
1
vote
1answer
241 views

Normalization the data before applying statistical test for large sample size

From my perspective, the reason p-value of a statistical test isn't useful in large sample scenario is because it will change according to the scale. E.g. let's focus on chi-square test. In a chi-...