# Questions tagged [large-data]

'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

441 questions
Filter by
Sorted by
Tagged with
90 views

### Huge sample sizes, tests, and deviation of assumptions?

I am performing Wilcoxon test (but in theory it can be any) and sometimes my sample sizes are huge. Even a single outlier may cause an extremely low p value. I am not interested in small effects at ...
26 views

### Data analysis in large samples

I am a graduate student in Mathematics and I am supposed to conduct a seminar concerning problems in data analysis, especially regarding large samples. I am looking for some resources providing ...
32 views

### What test to test difference between two independent groups of (very) unequal size?

My goal is to test the difference between two independent and non parametric samples of unequal size (3191 and 2209). One sample concerns questions and the other one responses and the data concerns ...
9 views

### Interpretation of the Hausman test (overidentification in relation to IV's) for large samples

I am using survey data with a huge amount of observations, such as the World Value Surveys. Large sample sizes are obviously very nice, but I have have encountered some downsides as well. To give an ...
65 views

### Learning from multiple very varied data sets?

Suppose we have a set of objects $X$ (e.g. individual humans). Suppose also that humans can be described by a set of (potentially very high-dimensional) variables $V_i$, (e.g. $V_1$ is a picture of ...
10 views

### Require suggestions on data handling

I have below tables in my database with record count as shown below I would like to join all these tables and make one final/main table. I tried the below approaches but nothing helped 1) Currently ...
11 views

### Question About Coming Up With Own Function for Distance Matrix (For Clustering)

Right now, I am currently working on implementing a clustering algorithm with millions data entries with regards to game users for a mobile game. A lot of the features I plan on using are unique to ...
163 views

### Difference between big data and high dimensional data

What do big data and high dimensional data mean? Is high dimensional data a special case of big data? What are the complications that arise in the analysis of high dimensional and big data each?
14 views

### Tests if k-samples are from same population are rejected for large N?

For a very large data set I have created a number of samples and try to verify whether the distribution of the variables in those samples is similar. I.e. I'm trying to verify that the sampling ...
42 views

### Statistical Significance in Table Ones

I have a question about statistical and clinical significance. In a non-clinical trials setting (e.g. observational), if the sample size is large for both groups (assume 2 groups for simplicity) ...
76 views

### Suggestions to cluster more than 300k observations

I am trying to perform an hierarchical clustering on data frame that contains 300k records 7 features (3 binaries and 4 continuous) in order to get insights on what looks like my dataset. I've chosen ...
26 views

### Detrending for a large dataset consisting of many group time-series

I have a huge dataset consisting of many individuals (~20000), each with a month of daily data. I am thinking of detrending those individual time series that are non-stationary but visually ...
80 views

### Multivariate and high-dimensional data, are they the same?

I read about the multivariate and high-dimensional data set. I found that the multivariate data is the data with more than 3 variables. In addition, the high-dimensional data is the data with a large ...
49 views

### Standard deviation for proportion with huge denominator

My question involves finding the standard deviation of a ratio when the denominator is huge. The standard formula $$\sqrt{\frac{p(1−p)}{n}}$$ yields a number that is almost zero. The context is ...
134 views

### When to prefer PCA over regularization methods in regression?

When dealing with the curse of dimensionality, regularization methods seem to be clear in their intuition. All "regularization" methods can be seen as a "squeezing" of one's variables towards ...
33 views

### Estimating probability density function of big amount of data coming from MC simulations

I am trying to estimate Probability Density Function (PDF) of a big amount of data ($1e^6$ , $1e^7$, and higher) coming from Mote Carlo (MC) simulation. My objective is to estimate the PDF (e.g. with ...
26 views

### Summary statistics for bipartite networks

I have a large bipartite network that I would like to summarise. So far, I have found the following summary statistics: Degree centrality Graph density Modularity Nestedness I have not found a ...
20 views

### Large scale SVM classification problem

Problem I am now working on a sentiment analysis task where the largest dataset involves 36 million custom reviews and associated sentiment (positive or negative). The feature extraction process is ...
147 views

### How to do exploratory analysis on very large dataset?

I have very large dataset stored in Google Cloud BigQuery. (the system dumps a table everyday. A table has approximately 25GB of size, and I have 2 months data = 60 tables = 60x25 GB) It is quite ...
13 views

### Difference between retraining on different portions of data and training initially on larger data set

I have a large data set that doesn't fit in memory and would have to use something like Keras's model.fit_generator if I would like to train the model on all of the ...
42 views

### Chi-Square Degeneracy for Large Sample

(Forgive my hand-waving explanation) When discussing anomaly detection methods (for example), one possibility is comparing the distance of a point from a centroid: Given 100 samples $X_1,...,X_{100}$ ...
15 views

### Compare and visualization of multiple output data files

I have 100 data files corresponding to 100 samples. For these 100 data files, I have processed data in 4 different ways to generate result output files. Out of the 4 results files per sample, I ...
155 views

### How to handle or impute large number of missing values?

I am trying to use this dataset to build a predictive model. The hubway.db file contains 3 tables. One of which is is bike_trips...
81 views

### Sampling Big Data for Machine Learning [closed]

In practice, how does one go about sampling a from big data set (eg. +/- 50 million distinct observations) to perform ML using Python? Most non-parametric models (e.g., SVM, ensemble models) start to ...
52 views

### How can statistics be used to avoid “Lending False Credibility To Decisions We've Already Made”

In light of this article Data Science Has Become About Lending False Credibility To Decisions We've Already Made published in Forbes, I would appreciate input from the statistical and data science ...
40 views

### Multiple Entries for Same Participant [closed]

I have raw data that I need to transform and unsure as to how. Manually doing it is out of the question due to the thousands of entries. I have the data in excel and I’m looking to analyze in SPSS. ...
134 views

### Fitting a multivariate Gaussian with extremely sparse samples

We have a multi-variate Gaussian distribution. For instance with 3 variables. The correlations between the variables are important! We are fitting it to data, however, the samples are such that each ...
50 views

### General question: How do you visualize/deal with a lot of predictors?

In STATS classes, one is always taught to draw a picture to look for outliers, to look for the distribution type, to look for patterns in general. However, when you have a dataset with a lot of ...
43 views

### Feature Selection with interactions in high dimensions

Is there any fast approach to find features considering interactions in many variables (~3000)? Many methods like RFE applying random forest would take very long. I tried MARS with degree=2 but it ...
26 views

### Clarification on quantification of Categorical variables

I have a countries column with 49 levels. I want to quantify it. If I run CATPCA on that column would i be able to get the quantified result. Since CatPCA is like PCA or factor analysis: it extracts ...
58 views

### Machine learning model underperformance on unseen data

This is a follow-up question to a question I had previously posted on this forum We conducted an experiment on 100 subjects and obtained a dataset that was used to train a machine learning model that ...
13 views

### Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
12 views

### Large datasets, deviations and noise/signal

The below excerpt mentions that 'large deviations' are more attributable to variance than to information. What does the author mean when he says 'deviations'? Deviations from what? And why would it be ...
34 views

### Comparing a distributions between large datasets

I have $2 < n < 10$ algorithms that I have applied to $10 < n < 500$ time series with ten different variables. The job of the algorithms is to produce a result that is as close as possible ...
497 views

### Does sampling from a large dataset lead to correct inferences?

Say we have some population, and we obtain a "representative" random sample of that population, $(y_i, x_i)_{i = 1}^n$, where $n$ is very large (millions) and $x_i = (x_{i1}, x_{i2}, ... x_{ip})'$ is ...
40 views

### How to interpret a given 2D co-variance matrix?

I am trying to solve a problem regarding revision for my Big Data module. I have two main questions. 1) Given a predefined co-variance matrix: A cluster of points is distributed in a two-...
79 views

### Calculate standard error for very large number of observations [closed]

I have a very large dataset (with > 2 million simulated values). I want to compute standard error for this dataset. To do that, I divide the standard deviation by square root of number of observations....
132 views

### What are some fast outlier detection methods for big data in R?

I have a large dataset (300,000 rows) for which there are clear outliers. Box plots of two of the DVs of interest reveal the presence of large numbers of outliers by the Tukey outlier detection rule (...
75 views

### Duplicated Rows in Mixed Data Type Clustering

I have a dataset which has ~200k rows and looks like the following - ...
32 views

### How to perform specific queries in weather data time series [closed]

I have time series data from several weather stations located in a specific area. The readings include a timestamp, the humidity and the temperature. The resolution of the data is quite high, about 6 ...
41 views

### t-test advice for simulations having multiple runs [duplicate]

Here's the gist of the problem - I am simulating n scenarios, each resulting from changing various simulation parameters. I want to demonstrate that I can offer a mechanism that would be able to ...
27 views

### How important is research on model selection methods in Statistics?

My question is nothing technical. I just wanted your opinion on how important is the model selection problem in the field of Statistics considering the age of big data. Are the current methods such as ...
111 views

### Where can I find high-dimensional (p>n) datasets? [closed]

I am looking for "high-dimensional" data for a course project. The requirements of an ideal dataset for me are: 1.$p>n$ (or at least $p> \sqrt{n}$), where $p$ is the number of variables and \$...
47 views

### residual plot for positive variable

I have a linear regression where the output variable is always greater equal zero with a nontrivial weight at exactly zero. The predictions of the model are also always greater than zero. What should ...
214 views

### How do I get the density of a region in a vector space?

I have a simple problem, which I think must have an easy solution. I have a vector space say with a 1000 dimensions for each vector. Now, I have a large number of sample vectors from this vector ...
33 views

### Deep learning with a lot of training data

I am building a bidirectional LSTM to do a sequential text-tagging task (particularly, automatic punctuation). Usually, the training is done in iterations, where in each iteration, the entire training ...
298 views

### Missing data imputation that can handle large data

I am looking for a reasonably scaling missing data imputation approach for big data (e.g. a well-scaling version of kNN - the standard versions we tried so far just ran out of memory) that fulfills ...
75 views

### How to combine multiple kernels of large sample datasets?

I have multiple large sample datasets in matrix format (each has 15000 rows and 5-50 columns) corresponding to different experiments. Each matrix contains the same number of samples(rows) but the ...