# May I use the whole dataset to prove the existence of a confounding variable in a machine learning framework if I don't use the labels?

I have a certain dataset that I am analyzing with machine learning techniques. I believe there is a certain variable (not used for training or testing the classifiers but is still known) that has an effect on the whole dataset used for ML.

I would like to fit a regression model on each variable in the ML dataset using the suspected confounding variable as the independent variable. After this, I would like to stratify the ML dataset so that the cross-validation folds are balanced with respect to the confounding variable.

Does this violate any ML assumptions? I know that using the test set at all is bad thing, if I were to look at the labels. However, I am not doing that. I am just comparing all data to a variable outside the ML dataset. Is this allowed? I believe so but just wanted to hear your opinions.

• Can you expound a little bit on what you mean when you say that the confounding variable "has an effect of the entire dataset"? – habu May 1 '15 at 13:51
• My interpretation is that the confounding variable has an effect on every variable in the dataset used for ML. So if the confounding variable is $c$, and the ML dataset is $X_{1}, ..., X_{N}$ then each $X_{i}$ is affected by $c$. – mmh May 1 '15 at 15:22

This sounds okay to me.

It seems that there are two stages to what you're trying to do:

1. Show that the dataset is significantly correlated with $c$.

2. Do whatever other machine learning stuff you wanted to do while accounting somewhat for $c$ by stratifying.

In order to do 1, I'd probably just run a cross-validated regression of some kind predicting $c$. The output of this will be just some estimate of the effect strength, not any model selection or anything like that. This will verify (or not) your hypothesis that $c$ matters.

If you confirm that $c$ is important, you can then run stratified cross-validation for whatever your original task was.

I don't see how any information from the test set, or any kind of multiple-comparisons type problem, would come from this.