How to evaluate a regression model?

I have a dataset with 190 samples. I'm training a regression model, using k-fold cross validation. I'm using $R^2$ for measuring the performance of the model.

However, sometimes, depending on the randomization process of the cross validation process, the $R^2$ is very bad. This happens because form a few samples, the results of the regression are very far from the expected ones, but, for most of the samples, the regression is very good.

We think that this happens because we have different probabilistic distributions in the dataset (coming from different environments with different characteristics). So, let us assume that we use a 10-fold cross validation method. And let us suppose that we have only 10 samples of a specific given probabilistic distribution. Sometimes, since the process is random, all these 10 samples are selected for being part of the test set and none of them are included in the training set. So, the model is not able to learn something about that probabilistic distribution. And, in this case, these 10 samples will produce a bad result for the model.

My question is: How can I evaluate my model fairly, in this situation? Should I control the 10-fold cross validation for ensuring that all the probabilistic distributions of the test set are represented in the training set? Or should I use other method?

• Welcome to CV. If the distributions you mentioned corresponds to well defined classes of the data, then you could use stratification to make sure that each fold is a good representative of the whole. Since you only have 190 samples, another option could be to do leave-one-out crossvalidation (e.g. set k=1). – matteo Sep 4 '18 at 18:38
• I agree with @matteo's first point. To expand on it, if you have different groups/classes with different distributions, you must account for that in your mode, not just in your R^2 calculations. Your model may be misspecified at present. – mkt Sep 5 '18 at 5:30