Model Stability Metrics and Solutions I encountered the term of "stability" in learning theory, well described on this wiki page and its references. The idea described in this page deals with the notion of how sensitive the learning algorithm is to changes in the learning set.
I would like to find some info on another notion of stability, which is how sensitive the prediction model to changes in the prediction set. I'll explain: Once We've already trained a model using the learning algorithm, and we have a model that won't change, I might want to run it periodically.
For example, a prediction model for the risk of developing a certain disease, which I run on a monthly basis. I might want to run the model on the same patient on subsequent months, while it's the same patient, but his/her data changes just a bit (a new lab test maybe).
I want to be able to measure and account for the change of predictions that is not due to real significant change in the patient, but to noise or other insignificant factors. I want to make sure the patient doesn't get an entirely different risk score for the same disease on each subsequent month.
I know about bias-variance tradeoff, regularization etc. and I have some other ideas of how to deal with this issue, for example:

*

*Different methods for lowering degrees of freedom of the model during training

*Monitoring the changes of individuals' predictions between months, and the reasons for these changes

*Feeding the model with the previous risk score of each person/sample

*Smoothing features

I would like to know if there's a professional term for this field of predictions stability for the same model. Also, are there any significant works published on this issue?
 A: I feel like you're asking the wrong question for what you want to achieve.
Bias-variance tradeoff exactly matches your description of the notion of stability. It also explains why you can't have both accuracy and consistency. The closest concept I can think of for what you want to achieve is robustness to noisy input, which can be achieved using data augmentation.
Firstly, I'll explain why asking for "notion of stability" is the wrong approach. Bias-variance tradeoff already answers that, but I'll describe it in a more generalized context in terms of implications. Suppose there is such a concept of stability, and there is some subjective procedure for measuring stability or variance of predictions to small changes in input. (Right off the bat, this procedure is unlikely to exist because what constitutes sensible changes in input varies for every problem and input/output format. It may exist for a highly specific problem type, but you haven't specified your problem type.) You go and test a whole lot of machine learning models, which are nominally ranked by accuracy or some other performance metric, and then you find that the most stable ones are the worst performing models. What use is that? Firstly you've created another problem of how to rank models by both difficulty and stability, and secondly, by default, the vast majority of known/popular models will have no support for decreasing stability (except by performing badly) so that information is barely useful. You also want to know when a change of prediction is dubious or wrong or borderline, which requires a meaningful confidence output from the model. Most models don't have a confidence output, and for those that do, confidence is generally unreliable itself; for a model to consistently and accurately gauge confidence it would have to be superior to a model that doesn't estimate confidence but is already consistent and accurate. That's begging the question and if you had good enough model in the first place then that itself solves the problem more directly.
You also have to define whether you prefer precise inaccuracy or imprecise accuracy. If the initial model prediction is wrong, and you've forced your model to prioritize consistency, then is being wrong forever really better than being giving indecisive results?
I'm going to suggest that what you're better off looking for one of:

*

*data augmentation

*a modified problem definition (like your suggestion "Feeding the model with the previous risk score of each person/sample") that encourages consistency

*a model that has some of the desired characteristics (such as a confidence output or where the "reasoning" behind a prediction can be accessed and interpreted), or

*a model that can be modified to have the desired characteristics, or

*a post-processing algorithm that modifies the outputs from a model based on past information, or

*some combination of the above

Data augmentation - This is where you generate artificially modified data from your original data, such as adding random noise to it, which will likely produce a model that is more robust to noise (and is therefore more consistent). There is a concern here in the context of diagnosis. If you add random noise, how do you know how much noise can be added without changing the true diagnosis? On one hand, it's fine if what you're saying is you want consistency even if it's factually unsound. On the other hand, if you care about factual accuracy then you need the expertise to generate sound noise-augmented data. Data augmentation aside, for a model to give stable predictions robust to noise, fundamentally there needs to be enough data.
