Linked Questions
14 questions linked to/from Online estimation of variance with limited memory
6
votes
2
answers
7k
views
Updating variance of a dataset [duplicate]
In an application, I deal will a large number of images from which I successively extract particular values to compute their means and variances. Because storing them all would mean storing Gigas, I'd ...
- 444
5
votes
3
answers
5k
views
How do I compute/estimate the variance of sequential data? [duplicate]
Say I have a (infinite) sequences like 1, 3, 2, 2, 1, 3 ...
I want to estimate their mean and variance of the sequence at time $t$.
But I won't have enough storage to keep all the data seen ...
- 16.8k
0
votes
1
answer
198
views
The variance of two running variances [duplicate]
Having 2 sets, and only this data for each one, the running variance, the sum, the running mean and the count. How can I get the merged variance of the 2 sets?
EDIT:
The values of the sets are being ...
- 103
0
votes
0
answers
242
views
How do I calculate the running variance? [duplicate]
I only get one value at a time in the system that I am developing. However whenever I get my first value I should already be able to calculate a variance.
Every time I get a new value I should ...
- 101
1
vote
0
answers
39
views
How does the nth sample impact Mean and Variance? [duplicate]
Given the Mean and Variance of $n$ samples $x_i$:
$$M_n=\frac {1}{n}\sum_{1}^{n} x_i$$
$$V_n=\frac {1}{n}\sum_{1}^{n}(x_i-μ_n)^2$$
How do Mean and Variance change, when we take into account one ...
- 1,250
21
votes
8
answers
4k
views
Why isn't variance defined as the difference between every value following each other?
This may be a simple question for many but here it is:
Why isn't variance defined as the difference between every value following each other instead of the difference to the average of the values?
...
- 368
11
votes
3
answers
2k
views
Is it possible to have Pearson correlation coefficient values $< -1$ or values $> 1$?
I am trying to calculate the Pearson correlation coefficient according this formula over a large dataset:
$$r = \frac{n\left(\Sigma xy\right)-\left(\Sigma x\right)\left(\Sigma y\right)}{\sqrt{\left[n\...
- 213
4
votes
3
answers
2k
views
Online algorithm for the mean square error
Given a dataset $\{(x_1, y_1), (x_2, y_2), \dots\}$, we can compute incrementally (or "online") the linear regression for those points.
In other words, given a new point $(x_i, y_i)$, we can ...
- 243
2
votes
0
answers
870
views
Change in standard deviation when a value is removed
Let's say a list of numbers $L$ has standard deviation $S$. Is there a formula for finding $S$ if I remove an element $l$ from $L$? Assume we know the mean of both $L$ and $L - l$.
- 143
3
votes
0
answers
610
views
Online algorithm to compute variance with a decay
Could somebody point me to an online algorithm that computes the variance, but gives a higher weight to more recent values?
1
vote
1
answer
247
views
Incremental update of Normal Distribution
There's a price time series $\{p_{t}, t=1..n\}$. Is it possible to estimate Normal Distribution for every data point $N_{t}(\mu_{t}, \sigma_{t})$ efficiently (like incrementally or online calculation)?...
- 447
0
votes
1
answer
99
views
What is the cumulants of a whole data in terms of the cumulants of its parts?
I have around 8 billion data points, and I need to calculate the distribution and the cumulants of this distribution.
However, due to technical restrictions, and time constraints, I can only ...
- 237
1
vote
0
answers
91
views
Equation to calculate t-statistic (independent samples)
I am looking to simulate the t-distribution from first principles.
In particular, I want to understand how the distribution arises by comparing the mean of sample A of size $n_A$ (taken from ...
- 451
0
votes
0
answers
47
views
How to approximate standard deviation with summarized stats [duplicate]
I'm trying to summarize incoming data to speed up some anomaly detection queries... but I'm not sure how to retain/use the stats I'm summarizing.
Here's my basic goal:
1 million events > 1 minute ...
- 101