Comparing classifier performance when using slightly different datasets

Question

Let's say I'm trying to predict whether tomorrow's temperature is higher than today's based on historical data (2 time series A and B). I've chosen XGBoost for the task. For model selection (hyperparameter optimization) and performance estimation I do the following:

1. split the dataset (ca 3000 samples) into train and test set (80/20).
2. do hyperparameter optimization on the train set using a cross-validation method suitable for time series, such as TimeseriesSplit in sklearn.
3. choose the best hyperparameters
4. retrain the algorithm with best hyperparameters on the entire training set
5. finally, estimate the generalization skill of the model by measuring accuracy on the test set

Next, I want to measure how the performance estimate changes when adding an extra variable, C, to the input set. Basically, to see if this feature has a negative or positive impact on classification performance. I get stuck when considering a method for performing this in a fair and non-biased way. Which steps should I follow?

My thoughts on fairness is that the hyperparameters need to be the same in both cases for comparison, correct? Or do I need to find he best hyperparameters for the [A,B,C] training dataset separately and then I repeat steps 3-5 for the second dataset?

Secondly, when evaluating on the test dataset, I can only compute 1 evaluation metric for each case (dataset). I feel this is not sufficient to compare performance as the results might depend on how the test set was sampled. How can I estimate variance for the metrics (e.g. accuracy)?

I've seen k-fold cross-validation being used for model evaluation. To my understanding, here the entire dataset (train+test) is split into folds (say 10), then 10 accuracy metrics are obtained from each fold and these are used to compute box plots for the 2 cases. Is this approach correct? It sounds strange to me to evaluate on roughly the same dataset the training was performed on.

Input on the specific steps I need to take to build a sound method are highly appreciated.

Answer 1

Your second question is directly related to the question of non parametric variance estimation. I advise you to look at the boostrap literature.

If you want more information on variance estimation in pratice maybe you can ave a look to there.

It is a small contribution to the debate and only a part of the answer.

Answer 2

Seems like you are interested in comparing 2 models. One model has 2 features/covariates: A & B. Your second model has 3 features: A,B,C. Both models have the same observations, just that one model has an extra feature "C". If those assumptions are correct continue reading for my suggestion.

Train these 2 models independently and perform 10 or 20 fold cross validation using the same fold spilt for the models. Thus each of your models would be trained and tested on the same "samples". Don't retrain the models on the whole dataset with the "best" hyperparameters as you suggested in Step 4. For each left out fold, record any number of performance metrics on the left out fold. Use an appropriate statistical test to compare the 2 models to determine if the second model is better.