Fair comparison of machine learning methods with different parameterization

Question

Let us assume the following scenario: The goal is to forecast the additional number of customers of a set of companies. To do this, you use a basic set of characteristics (e.g., company-specific characteristics and macroeconomic covariates) and are able to predict the figures quite well.

An employee at your company now says that she has collected a new dataset (e.g., 50 new features) that might be interesting to consider in the estimation process to further improve the performance of your machine learning models.

When you take the additional data into account, you notice better predictive performance, e.g. in terms of RMSE. However, the parameterization is also different because you have an additional set of 50 features.

In practice, can we say that the additional features make it possible to increase the performance metrics simply by comparing the two performance metrics? (Is this a fair comparison?) Or is it necessary to account for the different parameterization and the additional number of features?

Answer 1

1. As answered by @Dave, if your goal is out-of-sample prediction, you should not care at all that the two algorithms/processes have different set of features. The one with the better performance is the best one.

2. You may be interested in: "But do I really know that the second one is better than the first?" Then you are interested in statistical tests (or Bayesian analysis) if the two algorithms have different performances. In statistical testing, you need a set of measures/predictions quality for the first and for the second algorithms. You want to test if the prediction error of the second is statistically significantly different from that of the first. I would suggest a Wilcoxon rank sum if the two sets of measures were taken on different (new/out-of-sample) data, and Wilcoxon signed-rank if they were measured on the same new/out-of-sample data (even though the second algorithm used 50 new features of that data that the first did not use). If you can, use the signed-rank on the same data, it is a more powerful test). The usual is to conclude that the second algorithm is really "better" than the first if the p-value of the test is < 0.05.

3. There is a third question, which is "Is it worth it to use the second algorithm?" If both have the same cost (they are both developed, none of them is in use, the 50 extra features of the second model are free to collect, and so on) then there is no reason not to use the 2nd model even if the difference between them is not statistically significant. If there is some cost difference, for example, one is in use and the other is a prototype, there are costs in collecting the extra features, there are costs in maintaining the two different pieces of software, and so on, then one has to balance the gains in the RMSE of the second model with these costs.

4. Finally, your use of "fair" in the questions suggest to me (weakly) that you are interested also on things like "One model is a better explanation of the phenomenon that the other", because one does not use the extra 50 features. In this case, you are really in the in-sample domain, and you want some measure that balances performance quality with the complexity of the model. In this case AIC, BIC, Hannan–Quinn information criterion, are usual metrics to use (I do not know enough to suggest which one to use)

Answer 2

If the only thing you care about is prediction (and not estimating impact size of input factors), then what you put in the model is not that important as long as it 'works'. However, adding more features increases chances of some of them being correlated with outcome by chance, so you'll need to carefully cross-validate and ideally compare not only mean estimates for R2 RMSE etc, but their confidence intervals. I.e. if you improve R2 from 20% +/-3% to 25%+/-10% this may not really be an improvement and the model will be overfit and less stable on new data. It may be easier to bootstrap to find confidence intervals for all the metrics.