I would like to perform simple linear regression on a data set ($y_i = a x_i + b + \epsilon_i$) with $N \approx 50$. However, my residuals $\epsilon_i = \epsilon_i(x_i)$ exhibit heteroscedasticity as well as non-normality.

My first intuition was to perform some kind of robust regression to give less weight to outliers. However I run in a conceptual issue when it comes to cross-validation and testing due to the small sample size. When one has a lot of data, it is straightforward to divide it into train/cross-validation/test sets, estimate the weights on the training set, fixing hyperparameters with the cross-validation set, and evaluating the performance on the test set. However in the case where $N$ is small and residuals have non-constant variance, using 20% of the data (about 10 points) for testing purposes seems like a bad idea since outliers will most likely be overrepresented, skewing the "true" results of the data-generative process.

I have read that leave-one-out (LOO) cross-validation or $k$-folds cross-validation is useful for small data sets, but most discussions leave out the testing part entirely. Only doing cross-validation means that my model will find good hyperparameters but it will not appropriately generalize the data-generative process. What is the usual wisdom in statistics for such a situation?

  • $\begingroup$ What model do you have in mind? What hyperparameters are you trying to estimate? W/ a small dataset, you may need to work with a simpler model. $\endgroup$ – gung Jun 5 '17 at 19:46
  • $\begingroup$ @gung: I was thinking of using the Huber loss function, which is OLS loss for inliers and absolute value loss for outliers. The hyperparameter I had in mind was the residual threshold at which I start thinking of a data point as an outlier. $\endgroup$ – physguy Jun 5 '17 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.