# Questions tagged [large-data]

'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

565 questions
Filter by
Sorted by
Tagged with
19 views

### How to easily convert frequency data into raw data (large dataset) for t-test? [closed]

Statistics goal: Determine if the difference between two datasets is statistically significant. Dataset description: The data is available in the form of particle size (mm) v. particle count (...
196 views

### Can Wilcoxon be used in large sample with non-normal distribution?

I am doing my undergrad research, aiming to know the difference of before and after an intervention. our sample size is 37 which is already considered as a large sample right? However, when we test ...
148 views

### Detecting interactions in large logistic regression models

I have a dataset of a few million observations of a binary response with a low "Success"-probability of on average 1% to 2%. The dataset encompasses several categorical (~20 some with up to ...
• 2,708
1 vote
16 views

### Exploratory Factor analyses on large data sets

I have a question about using EFA on a large data set of survey questions. The goal is to form an index from over 200 items, and partly also as a form of dimension reduction (i understand PCA is also ...
• 11
1 vote
31 views

### Trust the graphs or go with Breusch-Pagan and White's tests for Homoscedasticity on large datasets? [duplicate]

I have a large dataset (n > 500,000) which I'm building a linear model with lm(PV1READ ~ PV1MATH + PV1SCIE + ST004D01T). Tests for Normality, No ...
• 159
28 views

### Decomposition of VAR(1) coefficient matrix

Consider the VAR(1) process $X_t = \Phi X_{t-1} + \epsilon_t.$ Is there a generally accepted decomposition for the coefficient matrix $\Phi$ that would decrease the degrees of freedom? My initial ...
• 81
27 views

### In the mgcv ::bam function in R, how can I constrain a two dimensional smooth to be monotonically increasing in both dimensions for large data?

I have a large dataset (1.3M rows) where I want to ensure that both Age and Duration increase monotonically for each by factor level (Male, Female). Here is the setup of the model: ...
• 41
33 views

### Clustering of large text datasets with unknown number of clusters

I have a list of hotel names which may or may not be correct, and with different spellings (such as '&' instead of 'and'). I want to use clustering in order to group the hotels with different ...
• 103
345 views

### Using a t-test to test effect size

In my line of work, I work with large data and often run stat tests to compare differences between groups. The problem I am facing is that if I use a $t$-test to measure any difference, the result ...
• 61
65 views

### p>>>n problem how to navigate?

I have a DNA methlation data for 32 samples. For each sample I have DNA methylation avaialble for >10000's of cpg bases (ie C nucleotides on DNA). I also have gene expression data from which I have ...
• 101
1 vote
84 views

### very low hosmer and lemeshow goodness of fit in logistic regression

I am currently working on my masterthesis. Therefor i want to perform a logistic regression (with logit link funtction) to predict the degree of encoded registrations in gp practices (coded ...
88 views

### Bayesian stats and multiple tests

Are Bayesian models subject to the same problems as frequentist ones, where we cannot run a bunch of different models due to Type I error? For example, let's say I have a large data frame on airplanes,...
• 89
1 vote
81 views

• 129
42 views

### Sampling and backwards selection

I'm working on a school project that involves performing backward stepwise regression as a form of feature selection. The dataset in question is 60k images with 700 total columns and is much too large ...
• 389
1 vote
53 views

### Small number of positives in a large dataset

I have a panel dataset with a very large number of observations 300,000. I am testing to see if a dummy variable is positive and significant using regular OLS. I have only about 1500 obs where the ...
1 vote
23 views

• 6,307
109 views

### feature selection within large dataset

I have a dataset containing more than 1000 predictors and I would like to do the feature selection. The features belong to several big categories(geographical factors, customer information....etc) ...
1 vote
42 views

### Is it safe to drop a few rows of data if working on a big dataset

I am currently working on big dataset. There are a few columns which are ordinal categorical data. In order to simply the dataset, I decided to change them into numeric. However, there are missing ...
361 views

### Normality test vs Gauss Markov assumption for panel data

I am doing fixed effect regression after conducting hausmann test on panel data. I received significant results in line with what's expected for my model. My data set has around 6000 observations and ...
32 views

### Imputation for Big Data [duplicate]

I have a dataset with 78 observations and 25000 variables.I am trying to apply logistic regression with shrinkage methods(penalties like SCAD,MCP,LASSO).The problem is that the commands ncvreg(ncvreg ...
• 31
2k views

### Bias Variance tradeoff in neural networks

Large neural networks have low bias and high variance. Training on large datasets greatly reduces the variance allowing them to fit complicated functions. My question is why they seem to have much ...
• 207
1 vote
36 views

### Where to start? [closed]

I am very new to data science and machine learning. I need some advice on how to recognize long/short-term patterns/trends in a big data set (demand data), make predictions for future and make optimal ...
106 views

### How to perform lasso on a wide matrix? [closed]

I have a Matrix with almost 1000 samples (rows) and for each of this I have gene expression data for more than 16000 genes. I was trying to perform lasso with the ...