Is there any existing result that characterizes how training error changes (increases) when the training data's distribution gets slightly changed while the minimizer of the original training data remains the same? In particular, I am assuming the model $f$ is 'smooth' in some sense(e.g., Lipschitz continuous or differentiable). In this case, I imagine the loss must be bounded if the perturbed dataset is within some neighborhood of the original training set w.r.t. some statistical distance. But Lipschitz continuity is about individual input, I am looking for something w.r.t. the entire distribution.

On the other hand, is there any existing result that characterizes how does a slight shift in training data distribution affects the change in the minimizer of the training data? I think this question is related to the stability of the learning algorithm. I have found some work, but I'd appreciate it if someone can point out the work related to stochastic gradient descent.

  • $\begingroup$ This is so vague and general that it is difficult to imagine any such result could possibly apply. Perhaps you have a specific kind of problem and a specific procedure in mind? $\endgroup$
    – whuber
    Feb 3, 2023 at 15:58
  • $\begingroup$ @whuber tried to add some more details. $\endgroup$
    – Sam
    Feb 3, 2023 at 16:39


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.