Difference between distribution shift and data shift, concept drift and model drift

Question

Lately, I am seeing both terms used interchangeably in several scenarios.

• Joaquin Quiñonero in MIT press (NIPS), Dataset Shift in ML
• NeurIPS 2021 workshop in DistShift
• Model drift: Towards Data Science

Are there differences in the definitions? Especially between data shift and distribution shift?

Answer 1

I'm not aware of a precise and accepted definition of each of these terms which sharply distinguishes them. There is an excellent blog post on the topic here. But broadly speaking:

• Model drift: This refers to the general idea that in some cases model predictions deteriorate over time. I.e. the distribution of model predictions and distribution of true values drift apart from each other. This can happen for a number of reasons.

  • Concept Drift: This is drift due to the dependent variable. The distributions of data might be staying the same but the relationship between input and output has been altered. For example, in a model to detect fraudulent activity, there may be a change in the definition of what is considered fraudulent.
  • Data Drift: This is due to changes in the distributions of the input data. For example, using the fraud example again, we might see an increase in certain types of fraud which change the distributions of observations from what was seen in the training data.