Questions tagged [large-data]
'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)
568 questions
3
votes
0
answers
50
views
Why does a random term in a large GAMM model make the curves spiky and wiggly?
I am creating several GAMM models with similar structures to dynamically model an acoustic parameter across realizations from multiple subjects, who are included as random smooths in my models.
In the ...
- 31
-1
votes
1
answer
32
views
Packages for Big Panel Data Regression in R [closed]
Is it possible to run a regression on a panel data set with 10,000 objects ($N$), each with $2,000$ observations (time series length $T$)? If so, what package in R can handle this?
- 557
1
vote
1
answer
95
views
Missing data imputation in longitudinal data in R
I have very big dataset (around 10 million rows) with repeated measures of around 500 000 individuals, irregularly spaced through time. My final goal is to do IPTW and fit a weighted cox regression ...
- 33
2
votes
1
answer
362
views
Dealing with non-proportional hazards in a Cox model with many variables and a large dataset
Summary: I am trying deal with non-proportional hazards in a Cox model on a large dataset. My question is whether the proportional hazards assumption really does not hold? If no, is the second model ...
- 209
2
votes
0
answers
29
views
Using whole training set for choosing model
I am working on a classification problem with what I understand as a big dataset. I have first of all splitted it in my "train" dataset and the "test" one. (Actually I am convinced ...
- 21
0
votes
1
answer
49
views
How to easily convert frequency data into raw data (large dataset) for t-test? [closed]
Statistics goal: Determine if the difference between two datasets is statistically significant.
Dataset description: The data is available in the form of particle size (mm) v. particle count (...
- 11
6
votes
1
answer
259
views
Can Wilcoxon be used in large sample with non-normal distribution?
I am doing my undergrad research, aiming to know the difference of before and after an intervention.
our sample size is 37 which is already considered as a large sample right?
However, when we test ...
- 61
5
votes
2
answers
234
views
Detecting interactions in large logistic regression models
I have a dataset of a few million observations of a binary response with a low "Success"-probability of on average 1% to 2%. The dataset encompasses several categorical (~20 some with up to ...
- 2,843
1
vote
0
answers
30
views
Exploratory Factor analyses on large data sets
I have a question about using EFA on a large data set of survey questions. The goal is to form an index from over 200 items, and partly also as a form of dimension reduction (i understand PCA is also ...
- 11
1
vote
0
answers
32
views
Trust the graphs or go with Breusch-Pagan and White's tests for Homoscedasticity on large datasets? [duplicate]
I have a large dataset (n > 500,000) which I'm building a linear model with lm(PV1READ ~ PV1MATH + PV1SCIE + ST004D01T). Tests for Normality, No ...
- 159
0
votes
1
answer
30
views
Decomposition of VAR(1) coefficient matrix
Consider the VAR(1) process $X_t = \Phi X_{t-1} + \epsilon_t.$
Is there a generally accepted decomposition for the coefficient matrix $\Phi$ that would decrease the degrees of freedom?
My initial ...
- 81
2
votes
0
answers
44
views
In the mgcv ::bam function in R, how can I constrain a two dimensional smooth to be monotonically increasing in both dimensions for large data?
I have a large dataset (1.3M rows) where I want to ensure that both Age and Duration increase monotonically for each by factor level (Male, Female). Here is the setup of the model:
...
- 99
0
votes
1
answer
39
views
Clustering of large text datasets with unknown number of clusters
I have a list of hotel names which may or may not be correct, and with different spellings (such as '&' instead of 'and'). I want to use clustering in order to group the hotels with different ...
- 103
6
votes
2
answers
361
views
Using a t-test to test effect size
In my line of work, I work with large data and often run stat tests to compare differences between groups. The problem I am facing is that if I use a $t$-test to measure any difference, the result ...
- 61
2
votes
2
answers
69
views
p>>>n problem how to navigate?
I have a DNA methlation data for 32 samples. For each sample I have DNA methylation avaialble for >10000's of cpg bases (ie C nucleotides on DNA). I also have gene expression data from which I have ...
- 101
1
vote
1
answer
138
views
very low hosmer and lemeshow goodness of fit in logistic regression
I am currently working on my masterthesis. Therefor i want to perform a logistic regression (with logit link funtction) to predict the degree of encoded registrations in gp practices (coded ...
2
votes
1
answer
97
views
Bayesian stats and multiple tests
Are Bayesian models subject to the same problems as frequentist ones, where we cannot run a bunch of different models due to Type I error? For example, let's say I have a large data frame on airplanes,...
- 107
1
vote
1
answer
84
views
Large sample distributions
Suppose we have observations $x_1, x_2, \ldots, x_n$ where $n$ is very large. Now we standardize the observations as $$y_i=\frac{x_i-\bar{x}}{\frac{s}{\sqrt{n}}},$$ where $s=\frac{\sum\limits_{i=1}^n(...
- 217
0
votes
1
answer
66
views
Seeking software recommendations for efficiently analyzing large 3-Level data with generalized mixed effect model
I am currently working on analyzing a substantial dataset using generalized mixed-effect models. The dataset has a 3-level structure: 400,000 individuals nested within 500 neighborhoods, which are ...
- 3
2
votes
1
answer
750
views
How to Check Linearity Assumption in Logistic Regression with a Large Dataset?
I am working with a very large dataset that essentially covers the entire population of interest. I want to assess the linearity assumption between an independent variable and the log(odds) of the ...
- 63
1
vote
1
answer
96
views
How to measure the increase in violent crime between years?
I have crime data from my country, consisting of the counts of recorded crimes (split by various categories), as well as population data from census years. I want to measure if there has been a ...
- 23
0
votes
0
answers
35
views
Understanding PCA plot built on data normalized by two normalization methods
I tried two different normalization methods, generated the PCA plot above on the combined data, and colored the samples by the normalization type. Both normalization methods should give similar ...
- 163
1
vote
1
answer
283
views
Approximate Gower's dissimilarity measure
I have a very large dataset with mixed-type variables. When I apply the Gower's dissimilarity measure to obtain the distance matrix, it is running out of memory. Due to the large size of the data, it'...
- 153
1
vote
0
answers
49
views
Is generation/evaluation of probabilistic predictions on continuous data feasible for larger data sets in practice?
To better capture uncertainty about the phenomena that we model, probabilistic predictions seem to be a natural and common extension of point predictions.
Methods for evaluation of these predictions ...
- 451
1
vote
1
answer
27
views
Dubios Binomial Residuals pt2
This problem was originally asked in another post as a minimum viable problem, such that all the details were generalized away. Posters on there seemed to want more information, so this is the more ...
- 99
7
votes
2
answers
1k
views
Building the right GAM model. Struggling with the jump from lmer()
I have I have no experience with gam() but I think this is the right way to model my data.
Sample data:
...
2
votes
0
answers
59
views
Is bagging less useful in 'big data' settings?
In 'big data' settings where the number of samples $n$ may be very large (for fixed number of features), is bagging less or more effective at reducing variance? I heard the claim that it is less ...
- 1,531
1
vote
0
answers
28
views
Clustering on thousands of product feature clicks and pages viewed
I want to classify 120k customers into 5-6 clusters basis the product usage, say, hundreds of product features clicked and hundreds of product pages viewed. The data will be like a customer_id has ...
- 31
3
votes
1
answer
135
views
t-test on non normal data: type I/II error vs validity
First, I don't believe this is a duplicate post even though this topic has been brought up a million times. If it is, please point me to the relevant post and I will remove this one.
I am basically ...
- 147
2
votes
0
answers
61
views
ANOVA Assumptions on Big data
I wanted to know if one would require to check for the violation of ANOVA Assumptions before running an ANOVA model on a Big Dataset (size of the big dataset is 57 million rows)?
Thanks!
7
votes
1
answer
172
views
Posterior consistency for scale-mixture shrinkage priors in low dimension?
Consider the model [1]
$$y_n=X_n\beta_n+\epsilon_n$$
$$\beta_i|\sigma^2,v_i \sim \mathcal{N}(0,\sigma^2 v_i), i=1,\ldots,p$$
$$v_i \sim \beta^\prime(a,b)$$
$$\sigma^2 \sim \mathcal{IG}(c,d)$$
where $\...
- 129
0
votes
0
answers
49
views
Sampling and backwards selection
I'm working on a school project that involves performing backward stepwise regression as a form of feature selection. The dataset in question is 60k images with 700 total columns and is much too large ...
- 391
1
vote
1
answer
60
views
Small number of positives in a large dataset
I have a panel dataset with a very large number of observations 300,000. I am testing to see if a dummy variable is positive and significant using regular OLS. I have only about 1500 obs where the ...
- 19
1
vote
0
answers
23
views
Information criterion for cases where $n\gg k$?
I'm looking for an information criterion metric that effectively penalizes the number of parameters ($k$) in cases where we have huge sample sizes. In my particular case my sample size is $\approx80,...
- 123
0
votes
0
answers
271
views
XGBoost - worse accuracy when increasing sample size? Survival analysis on a large dataset
I've encountered the above problem and struggle to find any info about online.
I'm messing around with a large dataset that I unfortunately cannot share. It has ~600 features of which not all are ...
- 1
1
vote
2
answers
139
views
Dimension Reduction and Clustering on Big Dataset
I am currently working on a marketing analytics project and my dataset is rather large: 400K rows and 70 features which contains both continuous and binary variables.
I've tried using PCA and ...
- 11
2
votes
1
answer
630
views
Extremely unequal sample size in anova
I have 14,000 data sets, and I'm going to do Anova in 5 groups.
The problem is that the size of Group A is 10,000, but the size of Group B is only 300.
In general, I understand that the unequal of ...
- 21
0
votes
1
answer
122
views
Median values over time
I'm evaluating a large data set in pandas (over 4 million rows), and finding that the median values for a couple of my datapoints are completely unchanged over a period of three months.
For example, ...
- 103
6
votes
3
answers
1k
views
Alternatives to scatterplot for visualisation for large samples
I would like to visualise the following scatterplot in a different way that would make it more intuitive. The X axis is trading frequency and describes how many trades were conducted whereas the Yaxis ...
- 305
2
votes
0
answers
35
views
Recommendation for books about statistical data analysis with sparse data ( a lot of zeros)
Evident from the title, i am looking for books about the general statistical data analysis techniques such as hypothesis testing, etc but for large and sparse dataset.
2
votes
2
answers
209
views
How to analyse a data set with more than 500 variables
I have a data set with 1000 entities. For each entity, more than 500 dichotomous variables (0,1) are recorded. I have no idea how to analyse this data or how to get an overview.
My problem is not that ...
- 89
1
vote
1
answer
59
views
high-dimensional GLS
I'm looking for a fast and stable method to compute high-dimensional GLS estimator.
Specifically, let $\mathbf{A}$ be a $p^2 \times m$ matrix with full column rank ($rank(\mathbf{A})=m$), $\mathbf{H}$ ...
- 387
0
votes
1
answer
138
views
Use of multivariate Shapiro Test for large sample size
I want to check for multivariate normality to conduct a MANOVA with two dependent variables.
Can I use Shapiro-Wilk test for multivariate normality if I have a rather large sample size (N = 1200)?
Or ...
- 1
5
votes
1
answer
101
views
Expected length of vectors after orthogonalization
Suppose I take $k$ vectors randomly sampled from surface of unit sphere in $d$ dimensions.
$$v_1, v_2, v_3,\ldots ,v_k$$
I apply Gram-Schmidt orthogonalization (but not orthonormalization) to obtain ...
- 6,317
0
votes
0
answers
111
views
feature selection within large dataset
I have a dataset containing more than 1000 predictors and I would like to do the feature selection. The features belong to several big categories(geographical factors, customer information....etc) ...
- 11
1
vote
1
answer
51
views
Is it safe to drop a few rows of data if working on a big dataset
I am currently working on big dataset. There are a few columns which are ordinal categorical data. In order to simply the dataset, I decided to change them into numeric.
However, there are missing ...
- 11
0
votes
0
answers
401
views
Normality test vs Gauss Markov assumption for panel data
I am doing fixed effect regression after conducting hausmann test on panel data.
I received significant results in line with what's expected for my model. My data set has around 6000 observations and ...
0
votes
0
answers
32
views
Imputation for Big Data [duplicate]
I have a dataset with 78 observations and 25000 variables.I am trying to apply logistic regression with shrinkage methods(penalties like SCAD,MCP,LASSO).The problem is that the commands ncvreg(ncvreg ...
- 31
10
votes
4
answers
3k
views
Bias Variance tradeoff in neural networks
Large neural networks have low bias and high variance. Training on large datasets greatly reduces the variance allowing them to fit complicated functions. My question is why they seem to have much ...
- 207
1
vote
1
answer
36
views
Where to start? [closed]
I am very new to data science and machine learning. I need some advice on how to recognize long/short-term patterns/trends in a big data set (demand data), make predictions for future and make optimal ...