I would like to perform simple linear regression on a data set ($y_i = a x_i + b + \epsilon_i$) with $N \approx 50$. However, my residuals $\epsilon_i = \epsilon_i(x_i)$ exhibit heteroscedasticity as well as non-normality.
My first intuition was to perform some kind of robust regression to give less weight to outliers. However I run in a conceptual issue when it comes to cross-validation and testing due to the small sample size. When one has a lot of data, it is straightforward to divide it into train/cross-validation/test sets, estimate the weights on the training set, fixing hyperparameters with the cross-validation set, and evaluating the performance on the test set. However in the case where $N$ is small and residuals have non-constant variance, using 20% of the data (about 10 points) for testing purposes seems like a bad idea since outliers will most likely be overrepresented, skewing the "true" results of the data-generative process.
I have read that leave-one-out (LOO) cross-validation or $k$-folds cross-validation is useful for small data sets, but most discussions leave out the testing part entirely. Only doing cross-validation means that my model will find good hyperparameters but it will not appropriately generalize the data-generative process. What is the usual wisdom in statistics for such a situation?