Let say that I want to compare the performance of XGBoost vs NN, or NN vs NN, or even the same NN at different epochs for a regression task.

All algorithms are trained and evaluated on the exact same dataset.

My thought is to compare the distribution of the residuals i.e.: set up a hypothesis test such tha $\mu_{xgb} > \mu_{nn}$, or do a t-test,...

Here is an example I was working on ...

enter image description here

enter image description here

As you may see, both models are similar, both are non-normally distributed, but NN's have a larger variance. I did not know how to compare, so I selected a paired Wilcoxon Signed-rank test since it does not assume a normal distribution. As expected, p-value was really low that the median of XGBoost is less than the median of NN.

I have no idea if this is kosher - but I could not find anything online.

Also, I was very surprised of how biased both models are in regions with the most frequent data. In terms of linear regression models - both of them would be considered terrible models. I would think QQ-plots would be a better measure than i.e.: feature importance in the case of XGBoost, if we assume

$$y = f(x, w) + \epsilon$$

where $x$ is the input and $w$ are weights in both models.

  • $\begingroup$ I would say ttest would be good.remember it is looking at the sample average of the residuals, so the central limit theorem will typically apply. $\endgroup$ – seanv507 Jan 1 at 9:34
  • 2
    $\begingroup$ Can you please explain what is the actual question? Is "picking the best performing model in this dataset", "showing statistical significance in performance", "assessing generalisation performance"? Something else? For practical purposes for example, does the MSE is more relevant than MAE? $\endgroup$ – usεr11852 Jan 31 at 17:11
  • $\begingroup$ When precisely do you mean by: "All algorithms are trained and evaluated on the exact same dataset"? Was this done with separate train and test sets, as @JacquesWainer assumes in an answer, or on the entire data set with internal bootstrap validation (another valid approach if you have fewer than several thousand cases), or do these results simply represent the performance of models trained on the entire data set and then evaluated on the entire data set? How large is your data set? $\endgroup$ – EdM Feb 1 at 22:09
  • 2
    $\begingroup$ This sounds confused. The mean of residuals will be 0 in most cases. The question is how narrow can you make the dispersion around your predictor. $\endgroup$ – DWin Feb 2 at 1:29

Because my last answer was downvoted, I'm going to provide a full example.

You don't want to compare the residuals, you want to compare losses. Let's say that your regression looks like this

enter image description here

Let's compare two models on RMSE: a linear model and a generalized additive model. Clearly, the linear model will have larger loss because it is has high bias low variance. Let's take a look at the histogram of loss values.

enter image description here

We have lots of data, so we can use the central limit theorem to help us make inference. When we have "enough" data, the sampling distribution for the mean is normal with expectation equal to the population mean and standard deviation $\sigma/\sqrt{n}$.

So all we have to do is perform a t test on the loss values (and not the residuals) and that will allow us to determine which model has smaller expected loss.

Using the data I generated

>> t.test(loss2, loss1)

    Welch Two Sample t-test

data:  loss2 and loss1
t = -7.8795, df = 1955, p-value = 5.408e-15
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.2717431 -0.1634306
sample estimates:
mean of x mean of y 
0.3761796 0.5937665 

The mean loss of the gam model is 0.37 while the mean loss of the linear model is 0.6. The t test tells us that that if the sampling distributions of the mean did have the same expectation (that is, if the losses for the models were the same) then the difference in means would be incredibly unlikely to observe by chance alone. Thus, we reject the null.

A paired method might help, but usually we have so much data that the loss in power is really not a problem.

Does that clarify things?

| cite | improve this answer | |

The usual for of evaluating a regressor in machine learning is by evaluating the error on a different dataset from the one used in training. I will assume that your sentence "All algorithms are trained and evaluated on the exact same dataset." mean that there are 2 different data sets, one for training and one for testing, and they are the same for both regressors (XGBoost and NN for example). If you are using only one dataset for both training and testing (which is the usual for people that use linear regression and talks about Q-Q plots of residuals) please DON'T. Split your single dataset into say, 80% training and 20% testing.

The distributions of the residuals (which in machine learning are similar to errors - more on this below) of the testing set are irrelevant. There is no assumption that errors/residuals for the regressor are normal, so Q-Q plot are irrelevant. And so on.

The main difference between residuals and errors is that error are always positive - from a prediction point of view, predicting an y that is 5 above or below the correct value is equally wrong. There are two usual transformations from residuals to error - the MSE and MAE referred by user11852 - MSE takes the square of the residuals and MAE takes the absolute value.

Now you have to sets of measures of ERRORS (be it MAE or MSE or others) and you want the regressor with least mean or median error. Errors are seldom normally distributed, so a non-parametric test is the more canonical approach, but if you have thousand of data points in your test data set, then the central limit theorem as referred by Demetri Pananos apply, so you can use a parametric test.

Finally, there is an important point. The two sets of error measures (for each regressor) are PAIRED - that is each measure in one set has a correspondent measure on the other set. So you should use a paired test for that.

To summarize: a) use 2 data sets, b) compute the error from the residuals - use the square transform - it is more frequently used, c) choose the regressor with least median error and d) if you want to argue that the one you choose is "really" (or significantly) better that the other one, then use the Wilcoxon signed rank test (the paired version of the Wilcoxon rank-sum)

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.