Let's say I'm trying to predict whether tomorrow's temperature is higher than today's based on historical data (2 time series A and B). I've chosen XGBoost for the task. For model selection (hyperparameter optimization) and performance estimation I do the following:
- split the dataset (ca 3000 samples) into train and test set (80/20).
- do hyperparameter optimization on the train set using a cross-validation method suitable for time series, such as TimeseriesSplit in sklearn.
- choose the best hyperparameters
- retrain the algorithm with best hyperparameters on the entire training set
- finally, estimate the generalization skill of the model by measuring accuracy on the test set
Next, I want to measure how the performance estimate changes when adding an extra variable, C, to the input set. Basically, to see if this feature has a negative or positive impact on classification performance. I get stuck when considering a method for performing this in a fair and non-biased way. Which steps should I follow?
My thoughts on fairness is that the hyperparameters need to be the same in both cases for comparison, correct? Or do I need to find he best hyperparameters for the [A,B,C] training dataset separately and then I repeat steps 3-5 for the second dataset?
Secondly, when evaluating on the test dataset, I can only compute 1 evaluation metric for each case (dataset). I feel this is not sufficient to compare performance as the results might depend on how the test set was sampled. How can I estimate variance for the metrics (e.g. accuracy)?
I've seen k-fold cross-validation being used for model evaluation. To my understanding, here the entire dataset (train+test) is split into folds (say 10), then 10 accuracy metrics are obtained from each fold and these are used to compute box plots for the 2 cases. Is this approach correct? It sounds strange to me to evaluate on roughly the same dataset the training was performed on.
Input on the specific steps I need to take to build a sound method are highly appreciated.