I have a dataset with 190 samples. I'm training a regression model, using k-fold cross validation. I'm using $R^2$ for measuring the performance of the model.
However, sometimes, depending on the randomization process of the cross validation process, the $R^2$ is very bad. This happens because form a few samples, the results of the regression are very far from the expected ones, but, for most of the samples, the regression is very good.
We think that this happens because we have different probabilistic distributions in the dataset (coming from different environments with different characteristics). So, let us assume that we use a 10-fold cross validation method. And let us suppose that we have only 10 samples of a specific given probabilistic distribution. Sometimes, since the process is random, all these 10 samples are selected for being part of the test set and none of them are included in the training set. So, the model is not able to learn something about that probabilistic distribution. And, in this case, these 10 samples will produce a bad result for the model.
My question is: How can I evaluate my model fairly, in this situation? Should I control the 10-fold cross validation for ensuring that all the probabilistic distributions of the test set are represented in the training set? Or should I use other method?