I am doing repeated cross-validation/holdout, because the results vary a LOT depending on the splits.
To me it seems that your model (which you don't talk about but probably should) is unstable.
Instability is a problem when performing K-fold CV or repeated K-fold CV because it leads to high variance and incorrect conclusions. This answer from an influential scientific researcher sheds some light on the intuition behind the issue.
I would therefore argue to first reduce instability in your model, either by choosing a different model which is inherently more stable (e.g. L2 regularized regression or regularized SVM, not a decision tree..) and then do what you wanted to do.
You can definitely perform feature selection via cross validation, but you have to be very careful about the way it is done. In particular to avoid data leakage, and this may involve double (nested) CV
Quoting a Kaggle competitor:
If you use feature selection, and you cross-validate in order to estimate your error, you should include the feature selection step in your CV procedure.
A few threads where this is discussed: