# Identify a switch between probability distributions

Say I have two normal distributions with means $$\mu_1$$ and $$\mu_2$$ and standard deviations $$\sigma_1$$ and $$\sigma_2$$, respectively. A t-test reveals that the means are significantly different, but the two distributions overlap a fair bit, too. I am selecting randomly from distribution 1, but at some point switch to distribution 2 without my knowledge.

How would I identify the point at which the switch occurred? Presumably, t-tests could also help me identify when the mean has shifted significantly, but I don’t know how to divide my observations to apply the t-tests. (E.g., do I use my 10 most recent observations? 100?) Is there an alternative approach that can identify a switch between distributions? Would the accuracy of such a test be affected by the degree to which the distributions overlap?

• Please see threads tagged with change-point. – whuber May 14 '19 at 15:47

In my opinion, it's a tricky and relatively open question that has multiple possible answers/approaches.

I would do it this way: since you have the list of observed values from two normal distributions, I can calculate the p-value at each of the partition point of the list:

a = rnorm(100, mean=0, sd=1)
b = rnorm(50, mean=2, sd=1)
ab = c(a, b)
pvals <- c()
for (i in 2:148) {
pvals <- c(pvals, t.test(ab[1:i], ab[i+1:150])\$p.value)
}
plot(pvals)


You can also get the location of the smallest p values:

min(pvals)
[1] 2.591222e-13
which.min(pvals)
[1] 98

• There is a fairly large literature on this problem and many excellent solutions. You might start with en.wikipedia.org/wiki/Change_detection. The partitioning approach is a popular one, but the trick is to determine when a difference ought to be considered real. Comparing individual p-values is not a correct way to do that. – whuber May 14 '19 at 15:56
• @whuber Thanks for the comment! This is fairly new for me, would be a good learning opportunity :P – TYZ May 14 '19 at 15:57
• Is there anything in the reliability literature that would help with this question? It's certainly not my field of expertise (in fact, relatively clueless), but I'd imagine this is akin to process control problems and detecting when a process is "out of control" (or not)? Interesting solution you've posted indeed, but I think I'll read some of what whuber has posted too. – StatsStudent May 14 '19 at 16:04
• @StatsStudent I posted my approach without knowing that there's a field called "change detection" and the researches going on, so it's probably not that reliable as whuber mentioned in the comment above. – TYZ May 14 '19 at 16:10
• @YilunZhang I think bother are useful to examine. Even if the solution isn't optimal, I think it's useful to learn from. – StatsStudent May 14 '19 at 16:15

Mostly I've seen switch point analyses in the context of probabilistic programming. In particular I think this approached both in Probabilistic Programming and Bayesian Methods for Hackers and the PyMC3 documentation.

I think I can hack together something that would look similar to your problem to give you a good idea of where the switch is occurring and some sense of uncertainty.

import pymc3 as pm
import numpy as np

#Generating synthetic data:
n1 = 1000
n2 = 1000

m1 = 0
m2 = .5

sd1 = 3
sd2 = 2

dat = np.append(np.random.normal(loc=m1,scale=sd1,size=n1),np.random.normal(loc=m2,scale=sd2,size=n2))

# Building the model of the switchpoint:
with pm.Model() as model:
switchpt = pm.DiscreteUniform("switchpt",lower=0,upper=len(dat)-1)
idx = np.arange(len(dat))
mu = pm.math.switch(switchpt>idx, m1, m2)
sd = pm.math.switch(switchpt>idx, sd1, sd2)
obs = pm.Normal("obs", mu, sd, observed=dat)

# Sampling
with model:
step = pm.Metropolis()
trace = pm.sample(10000, tune=5000, step=step)

pm.summary(trace)
# 95% confidence interval is [987,1012]
# mean of 993


The confidence in this prediction is affected by standard deviations, means, and counts as one might expect. This effectively says if there is more overlap in the distributions they are harder to distinguish and thus require more samples.