1
$\begingroup$

Suppose that I have three datasets: a training set with no labels, a dev-set with labels, and a test set with labels.

My question is: Can I use the dev-set and the test set to train a model and predict the labels of the training set and then use the training set(with the predicted labels) to train a different model and test the model with the dev-set and test set? Why?

$\endgroup$
5
  • $\begingroup$ That smells of data leakage to me. $\endgroup$ Apr 21, 2021 at 15:49
  • $\begingroup$ @DemetriPananos Is it? $\endgroup$ Apr 21, 2021 at 15:51
  • 2
    $\begingroup$ You're using the test set to create labels for training data to create a model which will be evaluated on the test set? That sounds like data leakage to me. Use each data set exactly once for its intended purpose. Any deviations should be validated using something like the bootstrap $\endgroup$ Apr 21, 2021 at 16:02
  • $\begingroup$ @DemetriPananos Great, let me see. $\endgroup$ Apr 21, 2021 at 16:06
  • $\begingroup$ You could reasonably use the inputs for the test data, as long as you didn't use the labels, as that would be similar to transductive learning. However if you used the labels of the test data (which is what you want to predict) in any way in the fitting of your model that would introduce a bias when it is used for performance evaluation (potentially very substantial) $\endgroup$ Apr 21, 2021 at 17:26

1 Answer 1

4
$\begingroup$

I'm more confident this is just data leakage. Here, I generate an example where we label the training data using a model created from the test set. The model should have accuracy of 0.5 or close to, but when replicating the procedure the average test accuracy is much higher.

#create the test set

r = replicate(1000, {
  
  X = MASS::mvrnorm(n=100, mu=rep(0, 2), Sigma=diag(c(1,1)))
  eta = rep(0, nrow(X))
  p = plogis(eta)
  y = rbinom(nrow(X), 1, p)
  
  x = X[,1]
  w = X[,2]
  
  model = glm(y~x+w)
  
  
  # Create train set
  
  Xtrain = MASS::mvrnorm(n=1000, mu=rep(0, 2), Sigma=diag(c(1,1)))
  ytrain = predict(model, newdata = list(x=Xtrain[,1], w=Xtrain[,2]))
  ytrain = as.integer(ytrain<0.5)
  
  xtrain = Xtrain[,1]
  wtrain = Xtrain[,2]
  
  model2 = glm(ytrain ~ xtrain + wtrain)
  
  
  ypred = as.integer(predict(model2, newdata=list(xtrain=x, wtrain=w))<0.5)
  
  Metrics::accuracy(ypred, y)
  
})


This is very likely because of data leakage. I would not do this under any circumstances.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.