6
$\begingroup$

I am trying to select a regression model for datasets with a right-skewed outcome and where "outliers" are present (where outliers are very high values due to the nature of the data). The data are time-sensitive so for model building I am splitting the dataset in train and test sets based on time (no k-fold CV) and the aim is to get the best model in terms of predictive accuracy.

Given the skewness of the outcome and the presence of outliers, it seems that RMSE and R^2 are not very suitable. What would be the best error metric to use to evaluate different models (e.g. features and hyperparameters) in terms of predictive accuracy? The size of the dataset is small (in the range of 500 for training and 100 for testing) so even a tiny number of outliers can alter my evaluation results.

Improve this question
$\endgroup$
0

3 Answers 3

Reset to default
4
+50
$\begingroup$

I don't think the problem here is your metric; RMSE & R^2 are generally quite acceptable. And in general, deviations from normality are not a major problem (see discussion here). However, if you have a number of outliers, you will likely improve your model if you change the error distribution you are using to one that can exhibit skewness.

Or alternately, you could transform your data so that your residuals are subsequently more normally distributed. If your residuals are right-skewed, a log-transformation or a square root transformation might solve your problem.

Improve this answer
$\endgroup$
11
  • $\begingroup$ Thank you, I am comparing penalized GLMs with tree models. So to get the optimal model (GLM or not and hyper-parameters), would I still use RMSE on my hold-out test set or maybe some more linear metric such as MAE would be more effective given that the errors in extreme values are influencing the RMSE results? $\endgroup$
    – user90772
    Jul 21, 2017 at 13:52
  • $\begingroup$ @user90772 That depends on what you are trying to optimise for, which is a decision that requires your judgement of the subject matter. RMSE penalises large errors more than MAE, and that may or may not be a desirable property for your purposes. If you include more details about your specific project, we may be able to advise you on that. $\endgroup$
    – mkt
    Jul 21, 2017 at 14:45
  • $\begingroup$ I am looking to predict how much a user will spend on a website given his demographics. The data is quite skewed which I guess is the rule in such problems. Is there any bibliography on this subject? $\endgroup$
    – user90772
    Jul 21, 2017 at 14:51
  • $\begingroup$ @user90772 We need more information, I'm afraid. In some sense, you are asking us to judge what type of error you should 'value' more, when this is a a somewhat subjective choice. Basically, we don't know what your loss function is. $\endgroup$
    – mkt
    Jul 21, 2017 at 14:54
  • $\begingroup$ The aim is to offer some voucher which is a percentage of the predicted spend amount before the user buys. So if a user is expected to buy $50, we would offer a voucher of 5, if 1000, then 100 and so on. Since I am not a statistician, I would like to understand the reasoning rather than the solution itself. $\endgroup$
    – user90772
    Jul 21, 2017 at 15:01
3
$\begingroup$

Just spitballing here, but this seems like as much of an economics problem as a statistics one. Given your application, it seems like you want to pick the model that maximizes your "profit": what the user would spend without the voucher plus the additional spend due to the voucher less the cost of the voucher.

I'm assuming you are predicting spend and, based on predicted spend, classifying them into one of a small number of voucher categories. If so, misclassification costs you more for different types of spenders, depending on how many there are, how much they spend, and how sensitive they are to your vouchers.

If so, you could do something like the following:

  1. Write down an equation for profit from a user conditional on their predicted and actual spend category. Aka, what will the average person in a spend category spend given their assigned voucher / predicted spend.

$ \pi(s,\hat{s})$ = ...

  1. Write down total expected profits using the the share of people in each category (from historical data) and the probability of misclassification.

$ \pi = \sum_sPr(s)\sum_\hat{s}Pr(\hat{s}|s)\pi(s,\hat{s})$

Now you have a function for profit based on your probabilities of misclassification. You can use this to select a model. Like RMSE or other metrics, this weights different size errors more or less, but it's based on your specific situation rather than a mathematical function.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for the detailed reply. We are not using any classification but we plan to offer a dynamic value based on the expected spend (rounded), rather than classifying users in some pre-defined buckets. So, the higher the expected spend, the higher the voucher value (linear function). We are not considering profit at this point, just spend. $\endgroup$
    – user90772
    Jul 25, 2017 at 21:28
0
$\begingroup$

Since your response variable is spending, it might be plausible to assume it's log normal and use a log transformation, which could remove the skewness. I believe the distribution of income is often fit well by a log normal, so maybe spending is as well.

What do you mean by time sensitive? Do you have measurements through time? If so you might be dealing with a time series problem where your errors are not independent. This will invalidate things like standard errors for a typical regression. You may want to look into models that account for correlation through time.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.