1
$\begingroup$

In a 2-stage stacked classifier the first model takes the input data and outputs feature vectors, which are then fed into a second model as input. The second model learns the mapping between the output of the first model and the data labels.

But what mapping does the first model learn? It is given the input data, but how can it learn learn relevant features if it itself is not seeing the labels? There are no "correct" feature vectors that the first model can calculate loss off of.

Is it possible to propagate the loss from the second model to the first model somehow?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

In the common stacked classifier setup, the base learners train using the true labels; they do not receive feedback from the second model. And generally, they don't produce feature vectors, just a single prediction; it is those predictions from several base learners that produces the feature vector for the second model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I see, so every model in the stacked setup trains using the true labels, so they only differ in what they receive as input features? In this case I'm confused as to what the subsequent models are supposed to be learning; they're trying to predict of off imperfect predictions, wouldn't that just increase noise and error without clear benefit? $\endgroup$
    – xojfqa
    Commented Apr 5, 2022 at 16:01
  • $\begingroup$ Even the input features can be the same and you might get some benefit. Consider the simpler "voting" classifier: the meta-learner just tallies votes from the base models. That can improve the predictions when different base learners perform better on different subsets of the data. $\endgroup$
    – Ben Reiniger
    Commented Apr 5, 2022 at 16:12
1
$\begingroup$

Yes, it is possible, but you need a single model that does that.

the first model takes the input data and outputs feature vectors, which are then fed into a second model as input

This sounds like a description of a standard two-layer neural network. A neural network is an example of such a model where the latter layers learn from the outputs created by the previous layers and propagate the errors downstream.

Improve this answer
$\endgroup$
2
  • $\begingroup$ But how does error propagation happen in stacked models that are not layers in a neural networks? Indeed, I was using the term propagation because of my knowledge of how backpropagation works in neural networks, but I don't see how that is generalizable to arbitrary types of models. For example, how do you use the output of an SVM as input to a random forest or vice versa? $\endgroup$
    – xojfqa
    Commented Apr 5, 2022 at 15:10
  • 1
    $\begingroup$ @xojfqa there is no simple way to do that. $\endgroup$
    – Tim
    Commented Apr 5, 2022 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.