Questions tagged [large-data]
'Large data' refers to situations where the number of observations (data points) is so large that it necessitates changes in the way the data analyst thinks about or conducts the analysis. (Not to be confused with 'high dimensionality'.)
564
questions
0
votes
1
answer
413
views
Covariance estimation in a big data setting
I don't seem to understand how to estimate the unbiased covariance of a big dataset, $X_{n,p}$. Suppose this dataset is to large to fit in memory: I have a billion samples in a moderate number of ...
- 21
0
votes
0
answers
401
views
How to do linear regression on categorical independent variable in excel?
I am newbie in data analytics world and I am trying to do linear regression on super-store sales database.
The database contain fields like OrderID, OrderDate, ShipDate, ShipMode, CustomerID, ...
- 101
7
votes
1
answer
13k
views
Batch Learning w/Random Forest Sklearn [closed]
I have a data set of approximately 5 million rows and wanted to run a RandomForestClassifier. I ran my RandomForestClassifier with only 50 trees, I tried to use the fit function but I receive a memory ...
- 101
8
votes
1
answer
311
views
Detecting changes in large number of time-series that share seasonality
I have large number of time-series that are independent of each other, but share some seasonality patterns. I need to detect anomalies/changes (increased volume, change in mean), that appear in the ...
Tim♦
- 138k
0
votes
1
answer
574
views
Statistics Optimal Number of Clusters in massive mixed dataset
There are several questions about this problems but they are not as specific as this and none of them have ever been really answered.
Using R, I am working on a ...
- 99
4
votes
0
answers
381
views
How to interpret the periodic phenomenon in learning curve? [closed]
I am working on training a deep neural network (with pre-training) on millions of data recently. However, I found out that the loss shows a form of periodic phenomenon (about 60000 steps for 1 epoch), ...
- 41
0
votes
1
answer
99
views
How does Stata estimate regressions in with large data sets (given small matsize)? [closed]
The upper bound of Stata's $matsize$ is 11.000.
I need to 'manually' compute my variance-covariance matrix for a data set with 300k observations.
Thus, I'm wondering, how does Stata internally ...
- 1,085
1
vote
1
answer
176
views
Advice on applying Machine learning for high dimentional datasets
I am working with a data-set of around ~100000 observations(rows) and ~256 features(columns). Is there any recommendation for applying Machine Learning techniques on such a data-set efficiently ? ...
- 565
2
votes
0
answers
2k
views
Multiple imputation using MICE for large dataset (~4,000,000 observations) [closed]
I am planning to carry out multiple imputation on a large data set with around 4,000,000 observations and 32 variables and 10 interactions. Of these variables six have missing data which I need to ...
- 355
1
vote
1
answer
278
views
Time varying regression in high dimension: cheaper than the Kalman filter
Assume you have a linear regression model $Y=\beta X$ with a high dimensional (say 1 000 000 resulting from dummy coding) vector $X$. You want to use this regression to predict $Y$. But dependence is ...
- 8,507
4
votes
1
answer
2k
views
Feature selection for very sparse data
I have a dataset of dimension 3,000 x 24,000 (approximately) with 6 class label. But the data is very sparse. The number of non-zero values per sample ranges from 10-300 (approx) out of 24,000. The ...
- 263
14
votes
5
answers
11k
views
Why is gradient descent inefficient for large data set?
Let's say our data set contains 1 million examples, i.e., $x_1, \ldots, x_{10^6}$, and we wish to use gradient descent to perform a logistic or linear regression on these data set.
What is it with ...
- 1,550
18
votes
1
answer
26k
views
Can support vector machine be used in large data?
With the limited knowledge I have on SVM, it is good for a short and fat data matrix $X$, (lots of features, and not too many instances), but not for big data.
I understand one reason is the Kernel ...
- 36.9k
0
votes
1
answer
185
views
Can we predict time by using Machine Learning?
I am struggling with a Use Case. In that use case I have to predict the time(hour and minute) to reach user. I am unable to find which ML Algo to solve this use case.
Can any one help to solve.
Thanks
5
votes
1
answer
1k
views
Forecasting short time-series at scale
I am looking for a method of forecasting short time series. I need to make multiple such forecasts at parallel, so I need some simple method that scales to large data. My data looks like $n\times k+1$ ...
Tim♦
- 138k
0
votes
0
answers
238
views
Linear regression with over 15 million datapoints with several factors
I want to use linear regression on a very large dataset with costs corresponding to about 15 million entries. For each of the 15 million entries I have several independent categorical variables.
I ...
- 255
5
votes
1
answer
109
views
Reference showing that only deep learning algorithms benefit from using huge datasets
Andrew Ng in his deep learning course on Coursera.org states that there is a boundary on sample size where machine learning algorithms stop improving and such boundary is nonexistent for the deep ...
Tim♦
- 138k
1
vote
1
answer
34
views
Should I be running R to find the significance with such a large data set and if so how to obtain it for each category
I have a very large data set which is over 30 of millions of records. My CSV file looks like this:
...
- 13
1
vote
0
answers
275
views
What are the pros and cons to use non-parametric methods to estimate probabilities?
When we have large amount of data, using logistic regression may suffer from high bias, i.e., linear model can underfit/too simple for large amount of data.
I am thinking to use some non-parametric ...
- 36.9k
0
votes
0
answers
42
views
Tree Boosting performance in overfitted models
Working with datasets that have millions of examples and thousands of available features, it's my experience that gradient boosted trees have a plateau in validation performance, after which ...
- 41
0
votes
0
answers
180
views
Sample Size vs. p-values [duplicate]
I'm using Cox-Regression to analyse the risk of loan portfolios.
I found out that with increasing sample size, my regressors are getting more and more significant (p-values are shrinking) due to ...
- 375
1
vote
0
answers
53
views
Modelling approach for large data set with many small clusters and rare outcome
Hi, I am trying to find out whether a certain behavioral trait in birds is heritable and have ~ 10K 3-generation pedigrees of birds (N total birds ~ 200K). Not all pedigrees have 3 generations of data-...
- 305
1
vote
1
answer
501
views
What is the right way for hyper parameter tuning when do partial fit on chunk data?
I have a huge data file, so i can not read it in memory. I read it chunk by chunk, then fit it by using partial_fit( like as : SGDClassifier). So how can i do
hyper parameter tuning for my model ? I ...
- 150
1
vote
1
answer
364
views
Using PCA to perform feature selection when variability and correlation are the only selection critera?
I'm running some data exploration and have a large ish number of variables with varying degrees of correlation and covariance. I'd like to start throwing out some variables I don't "need" according ...
- 23
-2
votes
1
answer
297
views
Classification problem and Associative rule mining [closed]
Imagine, you are solving a multiclass classification problem with highly imbalanced class. The distribution of the classes is such that, you observed the majority class 99% of the times in the ...
1
vote
0
answers
40
views
Self-study: why does the MLE cause selection bias when select subset of variables from large-scale predictors?
In the paper “TWEEDIE’S FORMULA AND SELECTION BIAS” the author Bradley Efron said the MLE can cause selection bias, which confuses me a lot.
We suppose that the statistician observes some large ...
- 147
0
votes
1
answer
2k
views
How to validate k-means result [duplicate]
I'm doing anomaly detection on unsupervised data using k-means I got a result but I don't know how to validate my clustering result.
by plotting I can see my anomalies but how should I validate that ...
- 141
2
votes
2
answers
284
views
What is the difference between sampling statistics and big data statistics?
What does significance mean in a classical sampling setting vs a big data setting?
Suppose there is a population P of events (P being arbitrarily large), and I randomly sample 50 of them. Suppose ...
- 21
2
votes
0
answers
852
views
High-dimensional embedding similarity normalization
I've generated a big set of very high-dimensional embeddings (7300 words with 1700 dimensions each) from a QnA dataset. I'm trying to visualize them and what I do is to apply ...
- 21
1
vote
1
answer
1k
views
Threshold for kmeans anomaly detection
I'm learning the kmeans to find out anomaly from the dataset.
but I don't know how to set threshold.
I tried by the putting mean of the centroid to point distance but it's not working, half my record ...
- 141
0
votes
1
answer
3k
views
How to calculate the distance between cluster center and datapoint in K-means
I'm learning about the K-means model.
I'm going to detect the anomalies using the k-means for that I need to calculate the distance between centers and point how should I do that.
- 141
1
vote
0
answers
598
views
Two sample Kolmogorov-Smirnov test
I have multiple sets of large (n,m>1500) experimental data and I am comparing 2 of the datasets at a time. I thought the two sample Kolomogrov-Smirnov test would be a good way to see if my datasets ...
- 11
1
vote
0
answers
67
views
Multiple imputation in repeated measurements setting?
I'm working on a large database (2 000 000 observations = people, with 50+ variables), with the objective to estimate how a binary exposure E affects a binary outcome Y (a disease) using logistic ...
- 23
0
votes
0
answers
116
views
Best practices to handle large-scale binary classification
I have a large training file (~2.4GB, ~31k rows). Each row contains around ~40k binary features, and the label is binary as well.
I tried to use default sklearn logistic regression (loading dataset ...
- 987
1
vote
1
answer
315
views
Recommendations for textbooks covering current data mining/machine learning techniques for fraud detection?
I work in the health insurance field, but a general treatment of fraud detection methodologies would still be helpful.
So far I've discovered brief articles outlining particular techniques used in ...
- 6,104
2
votes
0
answers
222
views
Leave-one-subject-out CV for large dataset
I have a large dataset of biomedical data from $40$ subjects consisting of $4$ features and around $2500$ observations per subject. Observations are categorized into six classes, i.e. each observation ...
- 3,552
1
vote
1
answer
2k
views
How to get p-values in high-dimensional settings?
It is easy to get p-values from a linear regression and related methods (t-test, anova, logistic regression), but how can one get p-values in a high dimensional setting (p >> n)? I understand that ...
- 7,609
-1
votes
1
answer
2k
views
Problems of small training dataset vs. large test dataset
I am currently working on a problem where my training dataset is very limited in size (few thousands of rows). Any model I develop on this needs to predict the outcome on a dataset millions of rows in ...
- 659
0
votes
0
answers
163
views
two sample test for large dataset
I want to perform two sample test on large dataset (in my case, each sample size is around one billion, and each data point is single real number). Which statistical test should I choose?
The null ...
- 31
2
votes
1
answer
2k
views
How to calculate critical values for R for a large dataset
I have gone through various formulas but all were used for a small data set like 100, 200 rows, but I want to know how to calculate critical values of r for a large data set containing >5000 rows.
I ...
- 121
2
votes
1
answer
130
views
How to visualise a data in dashboard for huge number of variables?
Consider I want to visualise the sales figure of 10 products in a dashboard (I will use HTML5 and JS). My data contains product name, total QoQ (Quarter on Quarter) of sales, QoQ of sales in US, QoQ ...
- 153
7
votes
2
answers
1k
views
How can I quickly detect cheating variables in large data?
Suppose We have a data set with millions rows and thousands columns and the task is binary classification. When we run a logistic regression model, the performance a lot better than expected, e.g, ...
- 36.9k
1
vote
0
answers
82
views
Tuning deep models with dataset subsample
I have a quite big dataset (380k samples), and I am try to do model selection over my validation set (3K samples). Since to run a single model requires days, I have subsample my training set (taking ...
1
vote
3
answers
1k
views
Samples in decision trees
I am trying to do some binary decision trees with Python (scikit-learn), but my sample has a bad repartition : I have something like 100 000 data points with label 0 and 800 000 with the label 1.
So ...
- 11
0
votes
1
answer
84
views
Logistic model selection on large data [closed]
I have a large dataset (750k rows). I have a-priorly selected 25 clinically meaningful variables for model building. I would have used likelihood ratio to build my model if not for my large dataset. ...
- 855
2
votes
1
answer
549
views
How to display a Kaplan–Meier plot (survival graph) with hundreds of thousands of datapoints?
I made a library for displaying survival graphs. I am currently struggling with figuring out how to meaningfully display data with 300,000 data points. There doesn't seem to be enough pixels in the ...
- 123
0
votes
0
answers
114
views
ARMA model estimation with very distant MA lags won't finish (Python pydse)
I have a restricted ARMA(24029,24029) model with lags 6969, 14332 and 24029 (all lags with both an AR and MA term).
I have the following code that I run in python:
...
- 943
1
vote
2
answers
195
views
ACF for long time series, trying to identify ARIMA models
I start to work with a dataset with more than 3000 samples, and realize that ACF plot does not make sense with respect of confidence intervals, see figure.
My purpose is to fit an ARIMA on this, ...
- 224
1
vote
0
answers
46
views
Analyzing jagged multidimensional data
I've been looking into this for a while now, but I don't think I know enough of the terminology to phrase this well enough for Google (my apologies).
So essentially I'm looking at motion capture data ...
- 141
4
votes
1
answer
983
views
scikit-learn regression trees/forest with very large data sets
I want to feed my data set (>2TB) into the scikit-learn regression tree first, but already in the beginning I face the problem of 'out-of-core' since the features for training are bigger than my RAM. ...
- 61