1
$\begingroup$

Suppose I have several models, one of them $M$ has a 100% training accuracy.

So regardless of how a stack the models, the stacked model is just M. e.g. If I use a linear model to stack them, then the new model is just $\hat{y}_\text{stack} = 0\cdot 1 + 0 \cdot \hat{y}_1 + 0 \cdot \hat{y}_2 + \dots + 1\cdot \hat{y}_M$

And I doubt $M$ is overfitting, because regardless of how I tune $M$, the test accuracy is only maximal when the training accuracy is 100%. In other words, I have not encountered a situation where training accuracy is increasing while the test accuracy is decreasing.

Is it impossible to stack models in this situation?

edit: None of the models has a test accuracy of 100% (but $M$ scores the highest), hence the need for stacking.

$\endgroup$
5
  • $\begingroup$ If you have already a model with 100% accuracy what's the advantage you want to obtain with stacking? $\endgroup$
    – GGA
    Jan 27, 2017 at 7:43
  • $\begingroup$ @GGA To improve test accuracy $\endgroup$ Jan 27, 2017 at 7:46
  • 1
    $\begingroup$ Wouldn't be the same to use only the model with 100% accuracy? What do you need to improve? $\endgroup$
    – GGA
    Jan 27, 2017 at 7:48
  • 1
    $\begingroup$ You could limit the coefficients of the linear model to a base of 0.1 and a maximum of 0.9, effectively saying that you trust each classifier a little (min 0.1) and no classifier completely (max 0.9). $\endgroup$ Jan 27, 2017 at 8:47
  • 1
    $\begingroup$ Use the validated statistics, training statistics are useless. $\endgroup$
    – Firebug
    Apr 4, 2017 at 0:22

2 Answers 2

1
$\begingroup$

You don't say anything about your data or the nature of the problem you're trying to solve, but on most real-world problems getting 100% accuracy is a red flag that you need to look deeply at your results and process.

100% means your model explains so much that there's NO UNEXPLAINED ERROR. NONE. If the underlying "data generating process" you're trying to impersonate is deterministic ... says a physics problem of power, acceleration, friction and distance ... getting to a perfect solution may make sense. But if the underlying data generating process is SUPPOSED to have some degree of randomness ... your results are telling you that you're nevertheless managing to perfectly nail even the "irreducible error." That is implausible.

That said, if you have a large representative sample of data to work with, and you're confident that your input predictors don't include any inappropriate leakage, and your M model is creating perfect predictions on an independent test dataset that is also reasonably representative of the (expected) future data that the deployed model will encounter ... then you have absolutely no reason to try an ensemble. Just use Model M.

$\endgroup$
0
$\begingroup$

Assuming your dataset was not specially defined/derived in some way (i.e. in a rare perfect case that your training set was perfectly classifiable), then you can try another train/test split and check if M still got accuracy of 100%.

The necessity of stacking model could be concluded once you build it and compare the performance with those base models.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.