Evaluating ML models is a fundamental task and subfield of the Machine Learning practice. On the other hand, I was not able to find any existing materials, guides, protocols, papers on how to proceed with evaluating/scoring complementary datasets (resulting multiple new features a.k.a feature set) when added to an existing model (and retrained with these new features added). This can be rather described as a with/without based comparative approach.
Let's say that there is some kind of ensemble model (e.g. catboost, lighGBM) trained on some $X_0$ and $y$ data with all the relevant testing metrics for that model type (classification or regression).
We receive some new feature sets, $X_1, X_2 ... X_n$. Our goal is to get an overview whether $[X_0 X_1], [X_0 X_2] .... [X_0 X_n]$ extended feature matrices test-split and trained with $y$ and tested afterwards shows some kind of significant improvement over the original (just) $X_0$ based model (meaning that the $X_i$ complementary feature set provided some valuable information for the model).
Generally the, first idea is to treat these similarly as in an iterative model improvement cycle, and just check the relevant model metrics (e.g. accuracy, f1, recall, precision, ACC for classification, R^2, RMSE, MAPE, MAE) if there is an improvement or not.
My issue is, that just looking at the metrics between the two model testing (with or without the new feature set), and doing something like a subtraction/percent difference seems to be somewhat blunt and not really informative on the significance of the improvement.
My other option was to come up with some aggregated, but prediction level comparative metrics based on the testing runs. For example with a classification type model, the number of testing predictions improving (which were wrong with only $X_0$), degrading (which were correct with only $X_0$) with the added new feature set ($[X_0 X_i]$ together).
- My question would be, if anyone can provide some good ideas on what kind of specialized metrics, methods, protocols, algorithms could be used for such comparative analysis and evaluation - to check whether adding a new feature set provides a significant improvement over the original model without that feature set?
- Can this done only in a ranking manner (ranking $X_1, X_2 ... X_i$ which feature set would be the best to add), or on some ratio based scale as well - comparing the level of improvement by feature sets?
- Any other insights, how to do this on a feature level as well would be appreciated. With ensemble models you mostly have the feature importance. But besides looking at those, and checking for which of the individual features out of the complementary feature set have received a high ranking importance, what can be additional metrics to see, which new features are valuable additions to the model (out of the new feature set)?
UPDATE
It might be much better for me to rather provide the business context, and that would describe the goals better.
My whole problem is related to the data acquisition/procurement process. A company does have some kind of production level ML model working as it is - trained and tuned with some carefully selected feature set. Then that company decides to look for additional datasets on the market, to create new/complementary feature sets to further enhance the model performance.
These new datasets come as a product (collection of data, tables, fields) from data vendors, with a hefty price tag. The company has to evaluate some trial datasets from these vendors, to see if they are valuable enough to buy and augment the existing ML pipeline with those (after engineering features out of the specific raw dataset). Lets say, that this is not an investment, quant ML trading model we are talking about, so no (simulated) backtesting can be used to simply measure the model/data performance by calculating ROI. But still, there needs to be some kind of decision made to buy or not buy a dataset. The general outcome of the model performance improvement is hard to measure in monetary terms. Rather it is some kind of revenue increase from product quality improvement by product price increase and client demand increase, so more likely the decision will be made on the discretion of someone business leader. But still, some kind of quantitative metrics must be provided for this business leader to assist them on the decision, and answer the question of "How much would this dataset generally increase the model performance, if we would buy/subscribe to it?", and "How much is this dataset worth (for xy fee) compared to another dataset on the market? Which should we choose?"
I feel like just providing something like adding this dataset (and the engineered features) - would decrease RMSE by X percent, or increase precision by Y percent is not a good enough solution. There must be some kind of better comparative metrics for this, to assist the business leader to quickly gauge the "ML value" of the dataset.
