Linked Questions
15 questions linked to/from Removing extraneous points near the centre of a QQ-plot
37
votes
7
answers
25k
views
Statistical methods to more efficiently plot data when millions of points are present?
I find R can take a long time to generate plots when millions of points are present - unsurprising given that points are plotted individually. Furthermore, such plots are often too cluttered and dense ...
- 473
6
votes
4
answers
4k
views
Should I sub-sample very large datasets to run the Kolmogorov-Smirnov (KS) test?
I have two lists with many millions of values (one has 21,410,024,757 values and the other 10,561,427 values). When I run the two-sample KS test the p-value is 0, but many posts suggest that for such ...
- 241
6
votes
2
answers
6k
views
Reducing number of data points in excel while keeping the curve shape
I've been gathering data at work to determine the losses in some cables that connect rooms, and now I need to take that data (~1500 points) and, because of software limitations, reduce it to 72 points ...
- 69
2
votes
3
answers
2k
views
What information is required to aggregate percentiles
I have several thousand large datasets that are too big to fit into memory at once, so I need to keep them separate.
It is easy enough to get the count, mean, std dev, min and max for the whole ...
- 131
3
votes
1
answer
831
views
Median from two unmergable datasets
I am trying to calculate the "overall" median of a variable that is spread across two datasets. I have access to the raw data in each dataset but can't bring their raw data together. What ...
- 133
2
votes
2
answers
1k
views
Making a continuous distribution from a discrete histogram
I was handed a discrete histogram of a random variable $x$. How do I generate 2,000 continuous samples from the histogram which represent the original random variable $x$?
My first thought is to fit a ...
- 2,051
1
vote
2
answers
589
views
How to approximate histogram(f(x)) from histogram(x)?
I have a histogram of a variable x, and I would like to get the histogram of f(x). Let's just say the transformation function is ...
- 292
2
votes
0
answers
523
views
Data compression for graph plotting [closed]
I am using Google Charts to plot a large data set. The database contains one record for every two seconds; five minutes' worth of data yields 150 records (data points) and the result is acceeptable. ...
3
votes
1
answer
350
views
Remove points from QQ plot of genome-wide association study
I have results from a genome-wide association study, which is basically a bunch of univariate tests (akin to linear regression). Results include statistical significance of each test, and thus I want ...
- 131
0
votes
0
answers
414
views
How to Compare Data Distribution of 2 large datasets?
I want to compare two super large datasets (petabytes).
I have tried to use scipy ks_2samp kolmogoroc smirnov 2 sample test, but I get all the pvalues to be 0. I tried to downsample by random sampling ...
- 21
0
votes
1
answer
111
views
How do I re-generate my bootstrap statistic based on saved summary statistics from bootstrap
So I estimated a particular statistic $\Phi$ (custom made) by bootstrapping 1000 samples from the original dataset to generate 1000 different $\Phi$s. The issue is that saving those 1000 bootstrap ...
2
votes
1
answer
117
views
How to determine whether a sample from a known population is significantly biased?
I have a large dataset (the population) and a large subset of it (the sample) containing the same, continuous variables. The sample represents more than 90% of the population but is not random -- we ...
- 133
0
votes
0
answers
109
views
Storing a probability distribution without saving single values
I saw this question "Storing a probability distribution without saving single values" on stackexchange and thought it deserved a statistical answer.
Example Scenario
I could see this problem ...
- 2,051
0
votes
0
answers
89
views
Averaging ECDF vertically : Proof of convergence
Suppose we have a set A , we split into multiple disjoint subsets ai
We only have access to the ai sets , is there a way to compute the ECDF for the set A without looking at it ?
If for example we ...
1
vote
0
answers
30
views
Calculating descriptive stats for aggregate/binned data [duplicate]
I have several datasets that I wish to analyse and summarize - and I'm starting with the basic median, and percentile/quartiles to begin with (unless there are other approaches I should be considering)...
- 33