3
$\begingroup$

What is parameter instability and how can I measure it?

If my model is having a hard time to forecast out-of-time samples, could parameter instability or populational instability be the cause of it?

$\endgroup$
1
  • 6
    $\begingroup$ This is a good question, but may be hard to answer. I suspect you will find that different people will use these terms in different ways. Can you provide some context? Where did you see these terms used, eg? $\endgroup$ Commented Mar 7, 2018 at 19:06

1 Answer 1

1
$\begingroup$

I agree with @gung there are some terminology or definition issues needs to be clarified.

What you described "hard time to forecast out-of-time samples" can be viewed as a "overfitting" problem in machine learning literature. For example, if we fit a very complex model on small amount of data, it is very likely the model "overfit" the data given, and fail to generalize to out of time samples.

Here are a widely used examples for overfitting (from wikipedia). You can see the model is trying too hard to fit the given data perfectly. Because the data contains noise, the model is not really generalizable to out of time sample.

enter image description here

To measure if the model is overfitting, bais and variance of the model can be calculated. For OLS, an example can find here. Intuitive explanation of the bias-variance tradeoff?


In fact, the first thing comes to my mind by reading the question title is numerical stability. This means because computer can only process finite digits numbers, there are algorithms that work theoretically but not working in real world.

$\endgroup$
1
  • $\begingroup$ This is possible, but the question is vague, IMO. My suspicion (but it's really only a guess) is that the "parameter instability" is that the relationship / parameter values are changing over time, such that the original fit could have been fine, but future accuracy will be poor nonetheless. Likewise, "population instability" might be that the model is learned on 1 population (say, grumpy old men), but subsequently applied to a different population (say, young whipper-snappers). Or they might be something else entirely... $\endgroup$ Commented Mar 9, 2018 at 1:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.