6
$\begingroup$

Background

I have a dataset, each record in this set is represented by 2 different sets of features. Let's say feature set A and feature set B.

I have trained classifiers using feature set A and feature set B respectively, and feature A outperformed. Now I would like to ensemble these classifiers to see if it is possible to get better performance.

One of the solutions

The first strategy I have tried is applying a data fusion strategy. For each prediction of the classifiers trained using the feature set A/B, I recorded the prediction probability. Then I calculated the mean value of the prediction probability for each prediction. After that, I get the prediction results of the data fusion strategy, which outperformed both features set A and B.

Question

Here's a problem in the solution above. I treated feature set A and B equally when calculating the average values, but what I would like to try is giving the outperformed feature set (feature set A) a higher weight.

Here I find an answer from @ Bruno Lubascher (https://datascience.stackexchange.com/questions/65973/differences-between-feature-weighting-and-feature-selection). In which he mentioned a stacking-like solution:

Example: You have an ensemble model, where all the feature coming into this model are actually predictions from other models. You might weight the predictions of these other models based on their individual performance. Then, your ensembler takes predictions from good performing models with more weight than from those with poorer individual performance.

My question is if the weight would be "learned" during ensembling the classifiers. Or the weight was given manually according to the performance of the classifiers trained using different feature sets? Or are there any other ways to learn weight for feature A and feature B in my case?

$\endgroup$
3
  • $\begingroup$ why do you want to treat the 2 sets separately? maybe a boosting model with all the features might work as well no? $\endgroup$
    – Alberto
    Commented Aug 1, 2022 at 12:08
  • $\begingroup$ Hi Alberto, thanks for your comment. In my case, both feature set A and B are high-dimensional. I'm afraid of the curse of dimensionality. $\endgroup$
    – BigTeeth
    Commented Aug 1, 2022 at 12:13
  • $\begingroup$ You can certainly stack models using Meta-Model Input $\endgroup$ Commented Jul 10 at 13:07

1 Answer 1

11
$\begingroup$

You can certainly stack your classifiers: apply your classifiers to yield classification probabilities in-sample, then train another model on the ground truth, using the two probabilities as predictors.

Alternatively, you could assess the quality of your classifications, e.g., using proper scoring rules, and transform these into a weight. This is easier, and you don't need to fit a third model.

As Alberto notes, you could also just create a model that uses all features, without fitting two separate models.

In any case, note that unweighted combinations often outperform "optimally weighted" ones (the "forecast combination puzzle"). One explanation is that finding "optimal" weights introduces additional variance, which passes right through to your final predictions (Claeskens et al., 2016, IJF).

$\endgroup$
4
  • $\begingroup$ Hi Stephan, thanks for your detailed explanation. I'm curious about the second solution you mentioned. Is it possible, for example, using accuracy scores to assess the quality of the classifiers and transfer them into weights? If so, are there any "scoring rules" I can use? $\endgroup$
    – BigTeeth
    Commented Aug 1, 2022 at 12:22
  • 2
    $\begingroup$ @BigTeeth Those would be the “proper scoring rules” mentioned in the answer, the most common of which are log loss and Brier score. // Stephan and I are among the advocates on Cross Validated for evaluation the probability outputs rather than bucketing the outcomes into discrete categories based on some threshold like $0.5$. This goes against a lot of common practices in machine learning but is more statistically sound, as common links discuss. $\endgroup$
    – Dave
    Commented Aug 1, 2022 at 12:35
  • 1
    $\begingroup$ The scoring-rules tag wiki has a lot of information. $\endgroup$ Commented Aug 1, 2022 at 13:09
  • $\begingroup$ A variation on stacking is to give the stacker the predictions from the two classifiers and the original features. It will be more work and -- in this particular case -- may require feature selection, since the two feature sets are high-dimensional. This technique can improve performance if the base classifiers are better at somewhat different subsets of examples. I learned about this approach from: N. Erickson et al. AutoGluon-Tabular: Robust and accurate AutoML for structured data (2020) arxiv.org/abs/2003.06505. $\endgroup$
    – dipetkov
    Commented Aug 1, 2022 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.