1
$\begingroup$

I am new to stacking. I have a dataset with N samples and 7 tables corresponding to different data types, plus a binary label. Some tables have dozens of features, other have many thousands. I train 10 ML models (SVM, KNN, XGB etc.) separately on each table, excluding one sample and forming a prediction on the excluded sample. Repeating for each sample, I get a Nx70 '1st-level' matrix, where each feature is the prediction for a particular model on a particular table. Performances vary, up to about 0.8 AUROC. Now, when I train a random forest on this matrix, again in a leave-one-out fashion, I get about 0.94 AUROC, which is suspiciously higher than the state-of-the-art for my problem.

I am concerned because in some sense, the prediction of sample 1 is using the 1st-level prediction for sample 2, sample 3 etc., which have been formed during the first step, using also sample 1. However I cannot imagine how the label could propagate through the predictions.

I would like to test this on a dummy dataset but I am not sure how to design this experiment. The proper x-validation scheme (exclude one sample from the start) would take too long to compute on the real data (takes about 12 hours on my laptop, and would need to be repeated N times). Any comment or help welcome!

Thanks a lot

$\endgroup$

1 Answer 1

0
$\begingroup$

Yes it is a problem. You should use nested cross-validation, so that CV of individual models is performed within the training set, and the test set was not used for fitting any of your models

$\endgroup$
2
  • $\begingroup$ As I stated, training all the 70 models times N samples takes 12 hours. So with nested-cross-validation I am looking at N-1 times training with N-2 samples. N is about 250 for the moment, pretty diverse so it is hard to have good stratification, hence the leave-one-out scheme. $\endgroup$
    – SebDL
    Commented Aug 3, 2022 at 12:27
  • $\begingroup$ the test subject cannot be used for training of any of your models if you want to have valid results without double dipping. 10-fold or even 5 fold cv would be fine with n=250 imo. Also, with n=250 it shouldn't take 12 hours, so maybe just throw away slow models or find better implementations $\endgroup$
    – rep_ho
    Commented Aug 3, 2022 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.