Skip to main content

Questions tagged [large-data]

'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

Filter by
Sorted by
Tagged with
0 votes
1 answer
19 views

How to easily convert frequency data into raw data (large dataset) for t-test? [closed]

Statistics goal: Determine if the difference between two datasets is statistically significant. Dataset description: The data is available in the form of particle size (mm) v. particle count (...
Dana Tran's user avatar
6 votes
1 answer
196 views

Can Wilcoxon be used in large sample with non-normal distribution?

I am doing my undergrad research, aiming to know the difference of before and after an intervention. our sample size is 37 which is already considered as a large sample right? However, when we test ...
Chilenesa's user avatar
4 votes
2 answers
148 views

Detecting interactions in large logistic regression models

I have a dataset of a few million observations of a binary response with a low "Success"-probability of on average 1% to 2%. The dataset encompasses several categorical (~20 some with up to ...
g g's user avatar
  • 2,708
1 vote
0 answers
16 views

Exploratory Factor analyses on large data sets

I have a question about using EFA on a large data set of survey questions. The goal is to form an index from over 200 items, and partly also as a form of dimension reduction (i understand PCA is also ...
Ewen Tan's user avatar
1 vote
0 answers
31 views

Trust the graphs or go with Breusch-Pagan and White's tests for Homoscedasticity on large datasets? [duplicate]

I have a large dataset (n > 500,000) which I'm building a linear model with lm(PV1READ ~ PV1MATH + PV1SCIE + ST004D01T). Tests for Normality, No ...
pluke's user avatar
  • 159
0 votes
1 answer
28 views

Decomposition of VAR(1) coefficient matrix

Consider the VAR(1) process $X_t = \Phi X_{t-1} + \epsilon_t.$ Is there a generally accepted decomposition for the coefficient matrix $\Phi$ that would decrease the degrees of freedom? My initial ...
Ville's user avatar
  • 81
2 votes
0 answers
27 views

In the mgcv ::bam function in R, how can I constrain a two dimensional smooth to be monotonically increasing in both dimensions for large data?

I have a large dataset (1.3M rows) where I want to ensure that both Age and Duration increase monotonically for each by factor level (Male, Female). Here is the setup of the model: ...
Colin's user avatar
  • 41
0 votes
1 answer
33 views

Clustering of large text datasets with unknown number of clusters

I have a list of hotel names which may or may not be correct, and with different spellings (such as '&' instead of 'and'). I want to use clustering in order to group the hotels with different ...
user480840's user avatar
6 votes
2 answers
345 views

Using a t-test to test effect size

In my line of work, I work with large data and often run stat tests to compare differences between groups. The problem I am facing is that if I use a $t$-test to measure any difference, the result ...
baz's user avatar
  • 61
2 votes
2 answers
65 views

p>>>n problem how to navigate?

I have a DNA methlation data for 32 samples. For each sample I have DNA methylation avaialble for >10000's of cpg bases (ie C nucleotides on DNA). I also have gene expression data from which I have ...
Saad Khan's user avatar
  • 101
1 vote
1 answer
84 views

very low hosmer and lemeshow goodness of fit in logistic regression

I am currently working on my masterthesis. Therefor i want to perform a logistic regression (with logit link funtction) to predict the degree of encoded registrations in gp practices (coded ...
Justine Soetaert's user avatar
2 votes
1 answer
88 views

Bayesian stats and multiple tests

Are Bayesian models subject to the same problems as frequentist ones, where we cannot run a bunch of different models due to Type I error? For example, let's say I have a large data frame on airplanes,...
aeiche01's user avatar
1 vote
1 answer
81 views

Large sample distributions

Suppose we have observations $x_1, x_2, \ldots, x_n$ where $n$ is very large. Now we standardize the observations as $$y_i=\frac{x_i-\bar{x}}{\frac{s}{\sqrt{n}}},$$ where $s=\frac{\sum\limits_{i=1}^n(...
user671269's user avatar
1 vote
1 answer
63 views

Seeking software recommendations for efficiently analyzing large 3-Level data with generalized mixed effect model

I am currently working on analyzing a substantial dataset using generalized mixed-effect models. The dataset has a 3-level structure: 400,000 individuals nested within 500 neighborhoods, which are ...
Wernicke's user avatar
2 votes
1 answer
481 views

How to Check Linearity Assumption in Logistic Regression with a Large Dataset?

I am working with a very large dataset that essentially covers the entire population of interest. I want to assess the linearity assumption between an independent variable and the log(odds) of the ...
LeterPeko's user avatar
0 votes
0 answers
28 views

Can't fit glmer in R with 18,000 observations when two proportions are very close

I have a dataset that has 3 levels (institution/provider/1:1 matched pairs) with 18,000 observations. The matched pairs (subclass) were exactly matched within their institution and might receive ...
HML's user avatar
  • 11
1 vote
1 answer
76 views

How to measure the increase in violent crime between years?

I have crime data from my country, consisting of the counts of recorded crimes (split by various categories), as well as population data from census years. I want to measure if there has been a ...
ac19's user avatar
  • 23
0 votes
0 answers
26 views

Understanding PCA plot built on data normalized by two normalization methods

I tried two different normalization methods, generated the PCA plot above on the combined data, and colored the samples by the normalization type. Both normalization methods should give similar ...
Rohit Farmer's user avatar
0 votes
0 answers
41 views

Optimization of SOM grid and number of metaclusters flowSOM

I am currently attempting to optimize the dimension of the SOM grid, as well as the number of metaclusters for some flow cytometry data in R, using the well documented flowSOM package. From what I ...
h3ab74's user avatar
  • 133
1 vote
1 answer
213 views

Approximate Gower's dissimilarity measure

I have a very large dataset with mixed-type variables. When I apply the Gower's dissimilarity measure to obtain the distance matrix, it is running out of memory. Due to the large size of the data, it'...
Phoebe's user avatar
  • 41
1 vote
0 answers
44 views

Is generation/evaluation of probabilistic predictions on continuous data feasible for larger data sets in practice?

To better capture uncertainty about the phenomena that we model, probabilistic predictions seem to be a natural and common extension of point predictions. Methods for evaluation of these predictions ...
QMath's user avatar
  • 451
1 vote
1 answer
27 views

Dubios Binomial Residuals pt2

This problem was originally asked in another post as a minimum viable problem, such that all the details were generalized away. Posters on there seemed to want more information, so this is the more ...
Kenney's user avatar
  • 99
7 votes
2 answers
887 views

Building the right GAM model. Struggling with the jump from lmer()

I have I have no experience with gam() but I think this is the right way to model my data. Sample data: ...
Simen Leithe Tajet's user avatar
2 votes
0 answers
48 views

Is bagging less useful in 'big data' settings?

In 'big data' settings where the number of samples $n$ may be very large (for fixed number of features), is bagging less or more effective at reducing variance? I heard the claim that it is less ...
WeakLearner's user avatar
  • 1,501
1 vote
0 answers
27 views

Clustering on thousands of product feature clicks and pages viewed

I want to classify 120k customers into 5-6 clusters basis the product usage, say, hundreds of product features clicked and hundreds of product pages viewed. The data will be like a customer_id has ...
Nebulum's user avatar
  • 31
3 votes
1 answer
114 views

t-test on non normal data: type I/II error vs validity

First, I don't believe this is a duplicate post even though this topic has been brought up a million times. If it is, please point me to the relevant post and I will remove this one. I am basically ...
David Wang's user avatar
2 votes
0 answers
52 views

ANOVA Assumptions on Big data

I wanted to know if one would require to check for the violation of ANOVA Assumptions before running an ANOVA model on a Big Dataset (size of the big dataset is 57 million rows)? Thanks!
Akira Banerjee's user avatar
7 votes
1 answer
166 views

Posterior consistency for scale-mixture shrinkage priors in low dimension?

Consider the model [1] $$y_n=X_n\beta_n+\epsilon_n$$ $$\beta_i|\sigma^2,v_i \sim \mathcal{N}(0,\sigma^2 v_i), i=1,\ldots,p$$ $$v_i \sim \beta^\prime(a,b)$$ $$\sigma^2 \sim \mathcal{IG}(c,d)$$ where $\...
MrDi's user avatar
  • 129
0 votes
0 answers
42 views

Sampling and backwards selection

I'm working on a school project that involves performing backward stepwise regression as a form of feature selection. The dataset in question is 60k images with 700 total columns and is much too large ...
jmoore00's user avatar
  • 389
1 vote
1 answer
53 views

Small number of positives in a large dataset

I have a panel dataset with a very large number of observations 300,000. I am testing to see if a dummy variable is positive and significant using regular OLS. I have only about 1500 obs where the ...
Greg Barns's user avatar
1 vote
0 answers
23 views

Information criterion for cases where $n\gg k$?

I'm looking for an information criterion metric that effectively penalizes the number of parameters ($k$) in cases where we have huge sample sizes. In my particular case my sample size is $\approx80,...
Haliaetus's user avatar
  • 123
0 votes
0 answers
198 views

XGBoost - worse accuracy when increasing sample size? Survival analysis on a large dataset

I've encountered the above problem and struggle to find any info about online. I'm messing around with a large dataset that I unfortunately cannot share. It has ~600 features of which not all are ...
NJL's user avatar
  • 1
1 vote
2 answers
122 views

Dimension Reduction and Clustering on Big Dataset

I am currently working on a marketing analytics project and my dataset is rather large: 400K rows and 70 features which contains both continuous and binary variables. I've tried using PCA and ...
BigDawg007's user avatar
2 votes
1 answer
476 views

Extremely unequal sample size in anova

I have 14,000 data sets, and I'm going to do Anova in 5 groups. The problem is that the size of Group A is 10,000, but the size of Group B is only 300. In general, I understand that the unequal of ...
Youngjin Kim's user avatar
0 votes
1 answer
93 views

Median values over time

I'm evaluating a large data set in pandas (over 4 million rows), and finding that the median values for a couple of my datapoints are completely unchanged over a period of three months. For example, ...
Steve B's user avatar
  • 103
6 votes
3 answers
1k views

Alternatives to scatterplot for visualisation for large samples

I would like to visualise the following scatterplot in a different way that would make it more intuitive. The X axis is trading frequency and describes how many trades were conducted whereas the Yaxis ...
magisterludi's user avatar
2 votes
0 answers
32 views

Recommendation for books about statistical data analysis with sparse data ( a lot of zeros)

Evident from the title, i am looking for books about the general statistical data analysis techniques such as hypothesis testing, etc but for large and sparse dataset.
2 votes
2 answers
165 views

How to analyse a data set with more than 500 variables

I have a data set with 1000 entities. For each entity, more than 500 dichotomous variables (0,1) are recorded. I have no idea how to analyse this data or how to get an overview. My problem is not that ...
Concetta's user avatar
1 vote
1 answer
58 views

high-dimensional GLS

I'm looking for a fast and stable method to compute high-dimensional GLS estimator. Specifically, let $\mathbf{A}$ be a $p^2 \times m$ matrix with full column rank ($rank(\mathbf{A})=m$), $\mathbf{H}$ ...
user0131's user avatar
  • 357
0 votes
1 answer
116 views

Use of multivariate Shapiro Test for large sample size

I want to check for multivariate normality to conduct a MANOVA with two dependent variables. Can I use Shapiro-Wilk test for multivariate normality if I have a rather large sample size (N = 1200)? Or ...
Janni's user avatar
  • 1
5 votes
1 answer
87 views

Expected length of vectors after orthogonalization

Suppose I take $k$ vectors randomly sampled from surface of unit sphere in $d$ dimensions. $$v_1, v_2, v_3,\ldots ,v_k$$ I apply Gram-Schmidt orthogonalization (but not orthonormalization) to obtain ...
Yaroslav Bulatov's user avatar
0 votes
0 answers
109 views

feature selection within large dataset

I have a dataset containing more than 1000 predictors and I would like to do the feature selection. The features belong to several big categories(geographical factors, customer information....etc) ...
Charlotte 's user avatar
1 vote
1 answer
42 views

Is it safe to drop a few rows of data if working on a big dataset

I am currently working on big dataset. There are a few columns which are ordinal categorical data. In order to simply the dataset, I decided to change them into numeric. However, there are missing ...
Charlotte 's user avatar
0 votes
0 answers
361 views

Normality test vs Gauss Markov assumption for panel data

I am doing fixed effect regression after conducting hausmann test on panel data. I received significant results in line with what's expected for my model. My data set has around 6000 observations and ...
Stataregression0192's user avatar
0 votes
0 answers
32 views

Imputation for Big Data [duplicate]

I have a dataset with 78 observations and 25000 variables.I am trying to apply logistic regression with shrinkage methods(penalties like SCAD,MCP,LASSO).The problem is that the commands ncvreg(ncvreg ...
Tsio90's user avatar
  • 31
10 votes
4 answers
2k views

Bias Variance tradeoff in neural networks

Large neural networks have low bias and high variance. Training on large datasets greatly reduces the variance allowing them to fit complicated functions. My question is why they seem to have much ...
efthimio's user avatar
  • 207
1 vote
1 answer
36 views

Where to start? [closed]

I am very new to data science and machine learning. I need some advice on how to recognize long/short-term patterns/trends in a big data set (demand data), make predictions for future and make optimal ...
2 votes
1 answer
106 views

How to perform lasso on a wide matrix? [closed]

I have a Matrix with almost 1000 samples (rows) and for each of this I have gene expression data for more than 16000 genes. I was trying to perform lasso with the ...
Riccardo F.'s user avatar
0 votes
1 answer
113 views

Time series model in production - Re-train on the fly as as batch process?

Let's say I've a time series of phone calls per day over the last three years. I could train a model using exponential smoothing (e.g. HoltWinters) for predicting the future amount of phone calls per ...
Constantin Müller's user avatar
2 votes
1 answer
234 views

Large Pearson residuals

My aim is to see if there exists a relation between the variables "Category" and "Types", spec. if there is a tendency for a particular category to use a particular type. These are ...
kdwu892's user avatar
  • 23

1
2 3 4 5
12