4
$\begingroup$

At time T0: A linear model is deployed to predict outcome Y as a function of X with equation Y = m1*x + c1

At time T1: The underlying process drifts to a new mean, resulting in consistently positive errors, i.e the intercept c1 is not equal to the expected mean in the new conditions. Correcting c1 to c2 yields errors that are now normally distributed with a mean of 0

Question: What would be the formal process by which you would detect that the underlying mean has changed? Is there a technical term for this process and is correction of the intercept a valid approach?

$\endgroup$
2
$\begingroup$

1. A Bayesian approach:

You could model your changing intercept as a stochastic process $\{C_t\}$. A natural assumption for modeling stochastic process $\{C_t\}$ is that it's a Markov process, that the past is independent of the future given the present. Since you don't directly observe $C_t$, this would be a hidden Markov model.

You would update beliefs $C_t$ from the data using recursive Bayesian estimation (also known as a Bayesian filter). Every period you observe the vector $\begin{bmatrix} X_t \\ Y_t \end{bmatrix}$. Using Bayes rule, you combine (i) your prior beliefs about the distribution of $C_{t-1}$ (and how $C_t$ evolves) with (ii) your observed vector at time $t$ to obtain the posterior, the new beliefs about the distribution of $C_t$.

Some common approaches to Bayesian filtering are:

  • Assume normality all over the place and use the Kalman filter.
  • Take a more flexible, numerical/simulation approach and use a particle filter.

Bayesian filtering might look overwhelming the first time you see it, but it's conceptually quite intuitive once you get into it. The whole idea is just to recursively apply Bayes Rule. The tricky part (IMHO) is keeping track of all the variables, not getting lost in the algebra, and not making simple mistakes.

2. Other ideas:

Something else you might want to look up are statistical tests for a structural break.

Does intercept $c$ come from linear regression? You could estimate $c$ based upon some rolling window of the past $T$ time periods. It's a bit ad-hoc for an observation last period to get the same weight as an observation T periods ago while an observation T+1 periods ago gets 0 weight, but rolling windows can work reasonably well in practice.

$\endgroup$
  • $\begingroup$ In my case the model is used every hour, every day. After a few weeks the error starts to shift. $\endgroup$ – Arun Jose Apr 4 '17 at 12:53
  • 1
    $\begingroup$ @ArunJose The Kalman filter for example can be used to track the location of a rocket using noisy sensor data as it flies through the sky. If your intercept is a much slower moving process, it might be overkill. $\endgroup$ – Matthew Gunn Apr 4 '17 at 13:03
  • $\begingroup$ Your second approach seems a better fit given the actual deployment conditions. Would my correction factor simply be equal to the average error in the rolling window in this case? $\endgroup$ – Arun Jose Apr 4 '17 at 13:21
  • 1
    $\begingroup$ (+1) Note that a rolling average can use non-uniform weights. $\endgroup$ – GeoMatt22 Apr 4 '17 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.