1
$\begingroup$

I am working on a ML problem incorporating four data streams, each producing multiple features.

We would like to know if each of the data streams provides a significant addition to the model performance.

If a data stream contribution is low, and upkeep cost is high, we should cut it from our model.

I have been thinking about what feature-importance technique to use, but most techniques are focussed on single features, like "Leave-One-Feature-Out" importance calulation 1.

However, in this problem we only have 4 data streams, and thus only 15 possible configurations. This is why I am considering what is basically an exhaustive search: Training a model for each of 15 combinations and sort results by data stream.

It sounds a but brute to me, but is there a term/name for this type of approach? I would love to read more on this type of testing.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ When we do something like this with hypothesis testing (testing multiple features at once, not just one at a time), common names are "partial F-test" and "chunk test". Those terms and their ways of thinking might lead you somewhere helpful. $\endgroup$
    – Dave
    Mar 30, 2022 at 14:18

0

Reset to default