Updating set of probabilities for sampling with features importance

Question

I'm currently working on some algorithm and I'm kinda out of idea for a problem I'm trying to tacle.

Basically I'm trying to subsample the features of a dataset. I want to subsample that given this critera :

1. if the feature is relevant it has a high chance of being drawn
2. if the feature is not relevent it has a low chance of being drawn
3. if it is not yet know if the feature is relevant or not it has an average chance of being drawn

In fact I'm drawing a subsample my feature at random than build a model that gives me the feature importances then I'm drawing my features again with the critera I explained above and then repeat that until I'm satisfied with my model.

The only thing that I have left to do is to find the right "formula" that would ponderate well the score that will help me calculate the probabilities to draw my variables.

Given the nature of the problem I would also like that if a feature is relevant in a certain set of feature and not in another it evens out.

I'm really open for all ideas. Thank you.

Answer 1

Perhaps you could try genetic algorithm. You could denote each feature by a boolean value, say a = 0, b=1, c=1, d=0; you are only using b and c. When you train a model on x combination of features, this combination will have value given by a fitness function. If this combination did well, then this combination is more likely to selected for reproduction (crossover, mutation), and therefore, b and c are more likely to show up in the next generation, consequently a and d are less likely.