Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [large-data]

'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)

0
votes
0answers
12 views

Difference between retraining on different portions of data and training initially on larger data set

I have a large data set that doesn't fit in memory and would have to use something like Keras's model.fit_generator if I would like to train the model on all of the ...
0
votes
0answers
23 views

Chi-Square Degeneracy for Large Sample

(Forgive my hand-waving explanation) When discussing anomaly detection methods (for example), one possibility is comparing the distance of a point from a centroid: Given 100 samples $X_1,...,X_{100}$ ...
0
votes
0answers
10 views

Compare and visualization of multiple output data files

I have 100 data files corresponding to 100 samples. For these 100 data files, I have processed data in 4 different ways to generate result output files. Out of the 4 results files per sample, I ...
0
votes
1answer
40 views

How to handle or impute large number of missing values?

I am trying to use this dataset to build a predictive model. The hubway.db file contains 3 tables. One of which is is bike_trips...
0
votes
2answers
29 views

Sampling Big Data for Machine Learning [closed]

In practice, how does one go about sampling a from big data set (eg. +/- 50 million distinct observations) to perform ML using Python? Most non-parametric models (e.g., SVM, ensemble models) start to ...
3
votes
0answers
45 views

How can statistics be used to avoid “Lending False Credibility To Decisions We've Already Made”

In light of this article Data Science Has Become About Lending False Credibility To Decisions We've Already Made published in Forbes, I would appreciate input from the statistical and data science ...
0
votes
0answers
30 views

Multiple Entries for Same Participant

I have raw data that I need to transform and unsure as to how. Manually doing it is out of the question due to the thousands of entries. I have the data in excel and I’m looking to analyze in SPSS. ...
1
vote
1answer
24 views

Fitting a multivariate Gaussian with extremely sparse samples

We have a multi-variate Gaussian distribution. For instance with 3 variables. The correlations between the variables are important! We are fitting it to data, however, the samples are such that each ...
1
vote
2answers
43 views

General question: How do you visualize/deal with a lot of predictors?

In STATS classes, one is always taught to draw a picture to look for outliers, to look for the distribution type, to look for patterns in general. However, when you have a dataset with a lot of ...
0
votes
0answers
18 views

Feature Selection with interactions in high dimensions

Is there any fast approach to find features considering interactions in many variables (~3000)? Many methods like RFE applying random forest would take very long. I tried MARS with degree=2 but it ...
0
votes
0answers
15 views

Clarification on quantification of Categorical variables

I have a countries column with 49 levels. I want to quantify it. If I run CATPCA on that column would i be able to get the quantified result. Since CatPCA is like PCA or factor analysis: it extracts ...
1
vote
1answer
44 views

Machine learning model underperformance on unseen data

This is a follow-up question to a question I had previously posted on this forum We conducted an experiment on 100 subjects and obtained a dataset that was used to train a machine learning model that ...
0
votes
0answers
11 views

Strategy to analyze large ( 20 mill rows and 200 columns) to predict a single variable

I am curious to understand how data scientists attack exceedingly large datasets in order to build a regression model for y? How does one decide where to start from? Reduce a large number of columns ...
0
votes
0answers
12 views

Large datasets, deviations and noise/signal

The below excerpt mentions that 'large deviations' are more attributable to variance than to information. What does the author mean when he says 'deviations'? Deviations from what? And why would it be ...
0
votes
0answers
31 views

Comparing a distributions between large datasets

I have $2 < n < 10$ algorithms that I have applied to $10 < n < 500$ time series with ten different variables. The job of the algorithms is to produce a result that is as close as possible ...
6
votes
4answers
200 views

Does sampling from a large dataset lead to correct inferences?

Say we have some population, and we obtain a "representative" random sample of that population, $(y_i, x_i)_{i = 1}^n$, where $n$ is very large (millions) and $x_i = (x_{i1}, x_{i2}, ... x_{ip})'$ is ...
0
votes
1answer
24 views

How to interpret a given 2D co-variance matrix?

I am trying to solve a problem regarding revision for my Big Data module. I have two main questions. 1) Given a predefined co-variance matrix: A cluster of points is distributed in a two-...
0
votes
1answer
41 views

Calculate standard error for very large number of observations [closed]

I have a very large dataset (with > 2 million simulated values). I want to compute standard error for this dataset. To do that, I divide the standard deviation by square root of number of observations....
0
votes
0answers
71 views

What are some fast outlier detection methods for big data in R?

I have a large dataset (300,000 rows) for which there are clear outliers. Box plots of two of the DVs of interest reveal the presence of large numbers of outliers by the Tukey outlier detection rule (...
1
vote
1answer
41 views

Duplicated Rows in Mixed Data Type Clustering

I have a dataset which has ~200k rows and looks like the following - ...
1
vote
1answer
31 views

How to perform specific queries in weather data time series [closed]

I have time series data from several weather stations located in a specific area. The readings include a timestamp, the humidity and the temperature. The resolution of the data is quite high, about 6 ...
0
votes
0answers
16 views

t-test advice for simulations having multiple runs

Here's the gist of the problem - I am simulating n scenarios, each resulting from changing various simulation parameters. I want to demonstrate that I can offer a mechanism that would be able to ...
0
votes
0answers
21 views

How important is research on model selection methods in Statistics?

My question is nothing technical. I just wanted your opinion on how important is the model selection problem in the field of Statistics considering the age of big data. Are the current methods such as ...
1
vote
2answers
38 views

Where can I find high-dimensional (p>n) datasets? [closed]

I am looking for "high-dimensional" data for a course project. The requirements of an ideal dataset for me are: 1.$p>n$ (or at least $p> \sqrt{n}$), where $p$ is the number of variables and $...
0
votes
0answers
43 views

residual plot for positive variable

I have a linear regression where the output variable is always greater equal zero with a nontrivial weight at exactly zero. The predictions of the model are also always greater than zero. What should ...
0
votes
0answers
13 views

What are some methods to analyse large sets of data in effective and efficient ways?

I have large amounts of data for clients of a transport provider (think similar to a taxi) in and around New York City. The kind of information I have is: What area they are travelling to and from. ...
0
votes
0answers
47 views

Fast Approximate Sampling from Multivariate Normal Parameterized by Precision Matrix

I want to efficiently sample $x \sim N(\mu, \Omega)$ where $\Omega$ is a precision matrix (e.g., the inverse of the covariance. The challenge is that the dimension of $x$ is massive (~ 100K to 10M) ...
1
vote
1answer
140 views

How do I get the density of a region in a vector space?

I have a simple problem, which I think must have an easy solution. I have a vector space say with a 1000 dimensions for each vector. Now, I have a large number of sample vectors from this vector ...
0
votes
1answer
31 views

Deep learning with a lot of training data

I am building a bidirectional LSTM to do a sequential text-tagging task (particularly, automatic punctuation). Usually, the training is done in iterations, where in each iteration, the entire training ...
2
votes
0answers
204 views

Missing data imputation that can handle large data

I am looking for a reasonably scaling missing data imputation approach for big data (e.g. a well-scaling version of kNN - the standard versions we tried so far just ran out of memory) that fulfills ...
1
vote
0answers
43 views

How to combine multiple kernels of large sample datasets?

I have multiple large sample datasets in matrix format (each has 15000 rows and 5-50 columns) corresponding to different experiments. Each matrix contains the same number of samples(rows) but the ...
1
vote
1answer
46 views

From Big Data to a normal regression problem

My goal is to predict taxi demand depending on location and hour in NYC. I constructed a large dataset with ~19 million observations. However, it is computationally very expensive to perform ...
1
vote
1answer
117 views

Normalization the data before applying statistical test for large sample size

From my perspective, the reason p-value of a statistical test isn't useful in large sample scenario is because it will change according to the scale. E.g. let's focus on chi-square test. In a chi-...
0
votes
1answer
112 views

Appropriate strength test of the chi-square test for large and unbalanced data

I understand different statistical tools have their own pros and cons. I'm trying to find the most appropriate one for my situation. I have a large, unbalanced data set and want to implement the chi-...
2
votes
0answers
31 views

Texts on visualizing big data

I am looking for textbooks, papers or alternative material focusing on ideas and general principles for visualizing big data. By big data I primarily mean wide data, i.e. high-dimensional data, but I ...
2
votes
1answer
44 views

Extensions of LSTM for huge data

Consider dealing with a huge high frequency financial data forecasting, RNN/LSTM is a popular way to solve the task. But the problem is that say you have 1 million data points and you want to predict ...
4
votes
1answer
73 views

When testing multiple hypotheses, what does it mean when there are not enough extremes? [closed]

Suppose you are testing a large number of hypotheses, say a million. Unlike the usual situation where you have a lot of very small p-values, in this case all of your p-values are greater than 5%. ...
0
votes
0answers
92 views

When is testing for normality necessary for machine learning with Big Data?

When is testing for normality necessary for machine learning with Big Data? Please give examples or counter examples.
0
votes
1answer
30 views

Flag redundant categorical variables in a big dataset

I have a dataset with ~150 categorical variables and ~150k rows. It is expected beforehand that a number of the categorical variables will be either identical, or nearly so. I would like to code ...
1
vote
0answers
164 views

R | Large-Sample Fixed-Effects Interaction using plm

How can I model Interactions with fixed effects for a large sample using plm? I have a panel data set with > 100,000 observations and I am trying to model a dummy-interaction with one of two fixed ...
6
votes
1answer
310 views

How to report: large sample sizes (10000+), significant small difference (1%)

It is known that with large sample sizes, even a small difference could be statistically significant although it may not be practically meaningful. For my current analysis, the difference in the ...
54
votes
7answers
7k views

Industry vs Kaggle challenges. Is collecting more observations and having access to more variables more important than fancy modelling?

I'd hope the title is self explanatory. In Kaggle, most winners use stacking with sometimes hundreds of base models, to squeeze a few extra % of MSE, accuracy... In general, in your experience, how ...
0
votes
0answers
126 views

Why is the size of fitted truncated svd model is so big?

I have a dataset with tfidf matrix of shape (200000, 565000). I am fitting truncated svd of 500 dimensions from sklearn onto it and pickling the resulting svd object for later use. The pickle file is ...
-2
votes
1answer
46 views

next year prediction based on data of previous years using R

For my project in R, I took 5 years(2011-2015) of socio-economic data from different cities in the US, and using this data, I want to predict the amount of births in the year of 2016(I have the data ...
1
vote
1answer
38 views

Goodness of fit for two samples with uneven different sizes

I have 2 different samples and would like to test a hypothesis about some aspect of their distribution. The first sample has 11,000 observations but the second is just 150 outcomes. From graphically ...
1
vote
0answers
45 views

How much data is needed for good ROC?

I am working on 2 class classification problem. I have over 1 million data points for each class available and I am building the Receiver Operating Characteristic (ROC) curve to evaluate classifiers. ...
1
vote
1answer
39 views

Clustering large 3D dataset into many clusters

I have a 3D point cloud with several million points and I need to partition it into roughly 50k clusters. As the clusters have to be spherical, usually a drawback of k-means, k-means seems pretty ...
2
votes
2answers
66 views

Chi-square test of independence of 2 variables on a large sample

Is it safe to check the Chi-square test of independence with a sample of 25000 ? I have the answers of approximately 25.000 people of different countries, as frequencies (total sums for each question)...
3
votes
0answers
63 views

What are the current popular methods for Large Scale Gaussian Process Regression, and which of them are readily available in R?

Vanilla Gaussian Process Regression requires $O(N^3)$ multiplications for estimation, $O(N^2)$ multiplications for prediction and it uses $O(N^2)$ memory where $N$ is the sample size, so it's not ...
7
votes
1answer
351 views

Random Forest in a Big Data setting

I have a dataset with 5,818,446 lines and 51 columns, where 50 of them are predictors. My response is quantitative, so I am interested in a regression model. I am trying to fit a random forest to my ...