I have a certain dataset that I am analyzing with machine learning techniques. I believe there is a certain variable (not used for training or testing the classifiers but is still known) that has an effect on the whole dataset used for ML.
I would like to fit a regression model on each variable in the ML dataset using the suspected confounding variable as the independent variable. After this, I would like to stratify the ML dataset so that the cross-validation folds are balanced with respect to the confounding variable.
Does this violate any ML assumptions? I know that using the test set at all is bad thing, if I were to look at the labels. However, I am not doing that. I am just comparing all data to a variable outside the ML dataset. Is this allowed? I believe so but just wanted to hear your opinions.