I have a method for prediction in a regression problem with outliers. I'd like to make a validation of my approach. How to make it considering that I need to have outliers in train and test datasets?
You could write a code to do have a fair distribution of outliers in both training and test sets for your cross-validation. So cross-validation will not be totally random, but it will represent more the distribution of your data. Then validate your model on an independent test set that also contains outliers and have a deeper look at the prediction for these outliers (are they correctly predicted or not?) to see if you should adapt model training or your dataset.