2
$\begingroup$

Evaluating ML models is a fundamental task and subfield of the Machine Learning practice. On the other hand, I was not able to find any existing materials, guides, protocols, papers on how to proceed with evaluating/scoring complementary datasets (resulting multiple new features a.k.a feature set) when added to an existing model (and retrained with these new features added). This can be rather described as a with/without based comparative approach.

Let's say that there is some kind of ensemble model (e.g. catboost, lighGBM) trained on some $X_0$ and $y$ data with all the relevant testing metrics for that model type (classification or regression).

We receive some new feature sets, $X_1, X_2 ... X_n$. Our goal is to get an overview whether $[X_0 X_1], [X_0 X_2] .... [X_0 X_n]$ extended feature matrices test-split and trained with $y$ and tested afterwards shows some kind of significant improvement over the original (just) $X_0$ based model (meaning that the $X_i$ complementary feature set provided some valuable information for the model).

Generally the, first idea is to treat these similarly as in an iterative model improvement cycle, and just check the relevant model metrics (e.g. accuracy, f1, recall, precision, ACC for classification, R^2, RMSE, MAPE, MAE) if there is an improvement or not.
My issue is, that just looking at the metrics between the two model testing (with or without the new feature set), and doing something like a subtraction/percent difference seems to be somewhat blunt and not really informative on the significance of the improvement.
My other option was to come up with some aggregated, but prediction level comparative metrics based on the testing runs. For example with a classification type model, the number of testing predictions improving (which were wrong with only $X_0$), degrading (which were correct with only $X_0$) with the added new feature set ($[X_0 X_i]$ together).

  • My question would be, if anyone can provide some good ideas on what kind of specialized metrics, methods, protocols, algorithms could be used for such comparative analysis and evaluation - to check whether adding a new feature set provides a significant improvement over the original model without that feature set?
  • Can this done only in a ranking manner (ranking $X_1, X_2 ... X_i$ which feature set would be the best to add), or on some ratio based scale as well - comparing the level of improvement by feature sets?
  • Any other insights, how to do this on a feature level as well would be appreciated. With ensemble models you mostly have the feature importance. But besides looking at those, and checking for which of the individual features out of the complementary feature set have received a high ranking importance, what can be additional metrics to see, which new features are valuable additions to the model (out of the new feature set)?

UPDATE
It might be much better for me to rather provide the business context, and that would describe the goals better.

My whole problem is related to the data acquisition/procurement process. A company does have some kind of production level ML model working as it is - trained and tuned with some carefully selected feature set. Then that company decides to look for additional datasets on the market, to create new/complementary feature sets to further enhance the model performance.

These new datasets come as a product (collection of data, tables, fields) from data vendors, with a hefty price tag. The company has to evaluate some trial datasets from these vendors, to see if they are valuable enough to buy and augment the existing ML pipeline with those (after engineering features out of the specific raw dataset). Lets say, that this is not an investment, quant ML trading model we are talking about, so no (simulated) backtesting can be used to simply measure the model/data performance by calculating ROI. But still, there needs to be some kind of decision made to buy or not buy a dataset. The general outcome of the model performance improvement is hard to measure in monetary terms. Rather it is some kind of revenue increase from product quality improvement by product price increase and client demand increase, so more likely the decision will be made on the discretion of someone business leader. But still, some kind of quantitative metrics must be provided for this business leader to assist them on the decision, and answer the question of "How much would this dataset generally increase the model performance, if we would buy/subscribe to it?", and "How much is this dataset worth (for xy fee) compared to another dataset on the market? Which should we choose?"

I feel like just providing something like adding this dataset (and the engineered features) - would decrease RMSE by X percent, or increase precision by Y percent is not a good enough solution. There must be some kind of better comparative metrics for this, to assist the business leader to quickly gauge the "ML value" of the dataset.

$\endgroup$
8
  • $\begingroup$ This is similar in spirit to a F-test. $\endgroup$ May 12, 2022 at 19:13
  • 1
    $\begingroup$ Frank Harrell addresses something like this on his blog. $\endgroup$
    – Dave
    May 12, 2022 at 19:34
  • 2
    $\begingroup$ This sounds more like a business/marketing question than a ML one : ultimately, the relevant metrics are those used by your customers when they decide whether to buy your product. A few percent increase in accuracy for example could be hugely valuable to some customers and completely negligible to others - As a company you should make an effort to understand what are the real needs of your customers. $\endgroup$
    – J. Delaney
    May 13, 2022 at 15:04
  • 1
    $\begingroup$ Do your new feature sets $X_i$ really correspond to the same observations for the outcome $y$? In other words, are you (a) getting new columns for the same rows of data you already had, or (b) getting entirely new rows with new columns? It makes a substantial difference. In case (a) you can use one of the many well-known methods for feature selection. Case (b) is more idiosyncratic since it's hard to compare apples to oranges. $\endgroup$
    – civilstat
    May 17, 2022 at 1:06
  • 1
    $\begingroup$ @Betelgeux Thanks for clarifying. And sorry, I didn't mean to imply that standard feature selection alone is enough to justify a business decision -- just that it could be part of the solution. J.Delaney is right that you need to understand the business metrics that matter to your customers and your business leaders. Translate those metrics into money, e.g. "This metric would have to improve by X% in order to outweigh the cost of buying new dataset A." Then your post's model-comparison methods can help you decide "We do/don't have evidence dataset A would boost the metric by X% or more." $\endgroup$
    – civilstat
    May 17, 2022 at 14:53

1 Answer 1

1
$\begingroup$

1)

You are right that difference or percentage gain in (say) accuracy with and without the new features is not a meaningful (in most of the cases see below) measure. The problem is that change in accuracy (or any other classification measure - I am not sure about regression measures but I believe so also) does not "mean the same" depending on the value of the accuracy. A 0.05 gain in accuracy is very different when the accuracy went from 0.65 to 0.70 in comparison when it goes from 0.92 to 0.97. A 0.05 change is much "meaningful" or "harder" the higher is the accuracy (or AUC, or F1, or precision, or MCC, and so on). The same for a 10% increase from 0.7 to 0.77 or 0.9 to 0.99!

2)

There is no accepted measure of change of ML performance metrics that is equally meaningful regardless of the base value. From now on I am improvising...

The issue is what the potential client for the extended dataset has already decided and what you know he/she has decided. You are being ambiguous or indefinite about many things, and that is OK. It may be that you don't know what potential clients have already decided, or it may be that you do not want the CV community to know, what you already know.

The mysterious issues in your question are:

  • A: the algorithm that is going to be used (by the client)
  • M: the metric that is going to be used
  • Y: the target variable that is going to be used.

Let call one of this variables fixed if the client has already decided on his/her choice and you know it (we the CV community may not know but that is OK).

Let assume that Y is fixed. That means that the client will use variable Y1 and you known it; everybody in this domain uses Y1.

Let also assume that M is also fixed. Everybody in this area uses M1 on Y1.

Finally, let us assume that A is not fixed. Clients may use different algorithms. In this case, the client knows some range of reasonable values for M1 for that problem (Y and M fixed). In this case, I think the convincing evidence is to claim that for a mix of different algorithms, using the extended dataset cause a mean gain or percentage gain of $\delta$ in metric. The client has a feeling what a gain of $\delta$ in this problem (Y and M fixed) means, and can evaluate the utility of getting the extended dataset. It may be the case that for a particularly powerful algorithm, the gain is not that large, but for less powerful algorithms the gain may be larger. Given that the client does not know what algorithm she will use, your mean figure is the gain she may expect.

If M is not fixed, you can list the mean gain for the different algorithms for each of the reasonable metrics. My personal opinion is that it is not meaningful to aggregate across different metrics - that is take the mean of gain for accuracy and F1 and AUC. I would list the mean gain for each of these three metrics.

If Y is not fixed, as with the M not fixed case, I would report on the gains for a few different reasonable Ys.

In summary, I would run the base and extended datasets for different algorithms and report the mean gain (for each reasonable Y and M). The point is that if the client knows how difficult is the problem (either because she knows Y and M or because you listed some reasonable alternatives) the mean gain, or mean percentage gain will make more sense because she has a feeling for the base value.

3

What if A is also fixed. Then the client can be clearly convinced if you show the $\delta$ difference. In this case, you may also want to show that the difference is statistically significant. In this case you have to test significance of two alternatives $X_0$ against $[X_0,X_1]$ on a single dataset. The usual procedure in M is for classification is the 5x2cv (5 repetitions of a 2-fold cross validation) proposed by Dietterich, Thomas G. "Approximate statistical tests for comparing supervised classification learning algorithms." Neural computation 10.7 (1998): 1895-1923. There is a question on this in CV Dietterich's 5x2cv paired t-test for regression problems

With luck, you can show that the difference is statistically significant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.