"Continuity" of SVM as a function of hyperparameters Suppose we have some (large enough) labeled training set and use some exhaustive cross-validation technique (e.g. leave one out) for tuning hyperparameters of SVM (with some nonlinear kernel). Is it true, that if we replace one observation by some another, the best possible hyperparameters would not change much?
I use the following technique:
loss_function(parameters, data)
  mse = 0
  for i=1 to n:
    model = train(data[all_but_i], parameters)
    mse = mse + (predict(model, data[i]) - y[i])^2
  return mse

for which I minimize the loss_function given data as a function of parameters. Am I right treating this as LOO?
 A: A model that does not change much if a small part of the training data is exchanged against a few other training cases is called stable, and I'd extend this also to hyperparameters. 
You can check this type of stability or robustness e.g. by iterated/repeated cross-validation (though not by LOO).
Even the cross validation used for the tuning can give you valuable information about this if you do not average the results for the different surrogate models: do all CV splits agree on which hyperparameters are best? How much spread is there, are closeby hyperparameters producing closeby performance for all splits?  
Sanity checks: 


*

*Look at the distribution of performance results for the same hyperparameters. How large a difference do you observe compared to this uncertainty? Does this allow you to select an optimum? 

*Does independent testing of the chosen hyperparameters yields very similar performance results to the inner performance results obtained during the grid search? If not, this is indication of overfitting which may mean that the optimization was driven by noise rather than by real performance differences. In this situation, the optimization is also not stable. 

update: See here as a start about model stability measurements with iterated/repeated cross validation
If you need a paper: Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 390, 1261-1271 (2008).
DOI: 10.1007/s00216-007-1818-6
