I have a certain dataset that I am analyzing with machine learning techniques. I believe there is a certain variable (not used for training or testing the classifiers but is still known) that has an effect on the whole dataset used for ML.

I would like to fit a regression model on each variable in the ML dataset using the suspected confounding variable as the independent variable. After this, I would like to stratify the ML dataset so that the cross-validation folds are balanced with respect to the confounding variable.

Does this violate any ML assumptions? I know that using the test set at all is bad thing, if I were to look at the labels. However, I am not doing that. I am just comparing all data to a variable outside the ML dataset. Is this allowed? I believe so but just wanted to hear your opinions.

  • $\begingroup$ Can you expound a little bit on what you mean when you say that the confounding variable "has an effect of the entire dataset"? $\endgroup$ – habu May 1 '15 at 13:51
  • $\begingroup$ My interpretation is that the confounding variable has an effect on every variable in the dataset used for ML. So if the confounding variable is $c$, and the ML dataset is $X_{1}, ..., X_{N}$ then each $X_{i}$ is affected by $c$. $\endgroup$ – mmh May 1 '15 at 15:22

This sounds okay to me.

It seems that there are two stages to what you're trying to do:

  1. Show that the dataset is significantly correlated with $c$.

  2. Do whatever other machine learning stuff you wanted to do while accounting somewhat for $c$ by stratifying.

In order to do 1, I'd probably just run a cross-validated regression of some kind predicting $c$. The output of this will be just some estimate of the effect strength, not any model selection or anything like that. This will verify (or not) your hypothesis that $c$ matters.

If you confirm that $c$ is important, you can then run stratified cross-validation for whatever your original task was.

I don't see how any information from the test set, or any kind of multiple-comparisons type problem, would come from this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.