How long will it take to detect that the parameters in a normal distribution have changed, given online sampling?

Question

Let's say, for instance, you have normally distributed data that is coming in one at a time. At some unknown instance, the distribution parameters (mean and variance) change. Assuming that there is not infinite memory, what would be the best way to determine how many samples are need to estimate the new distribution?

In this case, I was trying to play with a toy example where the first 100 events are drawn from N(100,20) and then at some unknown time, events are drawn from N(120,5). Intuitively, I think that it would take much longer to realize the variance shift in the distribution (i.e. it would take longer to detect a change moving from a very wide distribution to a narrow distribution, vs it would quickly be detected if you had a very narrow distribution previously and then shifted to a large distribution), but I'm not sure what the proper way to show this would be.

Any advice is greatly appreciated. Thanks!

Edit: Also in this case, the mean and variance of both the pre and post break point are unknown. Thanks

Answer 1

From a Bayesian point of view: $$ P(\theta|X_{new}) = \frac{P(X_{new}|\theta)P(\theta)}{P(X)} $$ Or in words: the probability the parameters, $\theta$, are what we think they are, given the data we've just observed, $X_{new}$, is given by the probability of getting the data we've just observed given the parameters times our confidence in the parameters.

If there have been hundreds, or thousands, of samples supporting $ X \sim N(100,20) $, and the process is believed stable, then $P(\theta=[100, 20])$ will be close to 1. When new data, $X_{new}\sim N(120,5)$, comes in, the original parameters, $\theta = [100,20] $ still look like a reasonable fit and $P(\theta =[100, 20] | X_{new})$ remains high. It's not until a large number of samples from $X_{new}\sim N(120,5)$ are observed that our confidence in $\theta=[100, 20]$ decreases and we can say we detect a change. On the other hand, if our confidence in $\theta=[100, 20]$ is low to begin with, $P(\theta=[100, 20])$ will be small and $P(\theta=[100, 20] | X_{new})$ gets smaller much quicker and we can say we detect a change much earlier.

This raises the question: are we willing to sacrifice being wrong about a change in distributions in order to claim to detect such a change sooner?