Yes, my objective is prediction. I will train/validate/test doing K-fold cross validation and so on. But my problem comes from the fact that high correlated features make the models (weights) unstable which is usually not recommended

Does anyone know what to do if feature x1*x2 and feature x2*x3 are highly correlated?

For every 70% of the train set, I pass it through a classifier and take the features passing a feature threshold (I use weights or feature importance) and create a counter out of these lists. I use "SelectFromModel" function from sklearn.feature_selection package.

In step 3, I am splitting the train test into 70-30 X times. That means that at the end I do use all the train set to find the subset of features. In step 4, I do k-fold cv on the 80% train (i.e. all data points from step 3)

What I want is to maximize the left side of the ROC curve (as attached in my question) which corresponds to sensitivity at high specificity.

Isn't LinearSVC with penalty 'l2' and linear kernel the same as Ridge classifier?

And how do I choose the features from that pool of features? The ones that were repeated the most?

What you are suggesting is training on the bootstrap samples (with replacement I assume) and test on the full original data? or test on the remaining data (i.e. repeated train/test split but with bootstrap)?

You can also try Leave-pair-out, which is the same as the approach proposed but you also test on the removed datapoint