Error metric for regression with right-skewed data and outliers I am trying to select a regression model for datasets with a right-skewed outcome and where "outliers" are present (where outliers are very high values due to the nature of the data). The data are time-sensitive so for model building I am splitting the dataset in train and test sets based on time (no k-fold CV) and the aim is to get the best model in terms of predictive accuracy.
Given the skewness of the outcome and the presence of outliers, it seems that RMSE and R^2 are not very suitable. What would be the best error metric to use to evaluate different models (e.g. features and hyperparameters) in terms of predictive accuracy? The size of the dataset is small (in the range of 500 for training and 100 for testing) so even a tiny number of outliers can alter my evaluation results. 
 A: I don't think the problem here is your metric; RMSE & R^2 are generally quite acceptable. And in general, deviations from normality are not a major problem (see discussion here). However, if you have a number of outliers, you will likely improve your model if you change the error distribution you are using to one that can exhibit skewness. 
Or alternately, you could transform your data so that your residuals are subsequently more normally distributed. If your residuals are right-skewed, a log-transformation or a square root transformation might solve your problem.
A: Just spitballing here, but this seems like as much of an economics problem as a statistics one. Given your application, it seems like you want to pick the model that maximizes your "profit": what the user would spend without the voucher plus the additional spend due to the voucher less the cost of the voucher. 
I'm assuming you are predicting spend and, based on predicted spend, classifying them into one of a small number of voucher categories. If so, misclassification costs you more for different types of spenders, depending on how many there are, how much they spend, and how sensitive they are to your vouchers. 
If so, you could do something like the following:


*

*Write down an equation for profit from a user conditional on their predicted and actual spend category. Aka, what will the average person in a spend category spend given their assigned voucher / predicted spend. 


$ \pi(s,\hat{s})$ = ...


*Write down total expected profits using the the share of people in each category (from historical data) and the probability of misclassification.


$ \pi = \sum_sPr(s)\sum_\hat{s}Pr(\hat{s}|s)\pi(s,\hat{s})$
Now you have a function for profit based on your probabilities of misclassification. You can use this to select a model. Like RMSE or other metrics, this weights different size errors more or less, but it's based on your specific situation rather than a mathematical function.
A: Since your response variable is spending, it might be plausible to assume it's log normal and use a log transformation, which could remove the skewness. I believe the distribution of income is often fit well by a log normal, so maybe spending is as well.
What do you mean by time sensitive? Do you have measurements through time? If so you might be dealing with a time series problem where your errors are not independent. This will invalidate things like standard errors for a typical regression. You may want to look into models that account for correlation through time.
