Is it always bad to retrain your model to include predicted data?

I understand intuitively why this is a horrible idea - you assume your model is correct and then increase your number of observations which will likely result in a poor fit on future data.

I'm wondering if there is some mathematical/statistical property to describe this, or if there is any rare case where this may not be as fatal as I am thinking?

• Fundamentally, adding data generated by your model will not introduce any new information, and so can't possibly benefit (only by accident). Oct 31, 2013 at 0:22

2 Answers

I can give you a probabilistic/Bayesian interpretation of why this is not helpful. A probabilistic model for data $X$ and parameters $\theta$ is defined by a likelihood $P(X|\theta)$ and a prior $P(\theta)$. Now imagine I have some training data $X_\text{train}$ and want to make predictions about future data $X_\text{future}$, which means I need to calculate, or approximate $$P(X_\text{future}|X_\text{train}) = \int P(X_\text{future}|\theta) P(\theta|X_\text{train}) d\theta$$ where $P(\theta|X_\text{train})$ is the posterior. What you suggest is to sample predictions $X_\text{pred}$ from $P(X_\text{pred}|X_\text{train})$ (which can be represented in the same way as the above equation). However, since $X_\text{pred}$ is not observed you can integrate it away and your posterior on $\theta$ will be unchanged. Conditioning on $X_\text{pred}$ is therefore not a reasonable thing to do.

To speculate on the typical effect it might have: if you sample from $P(X_\text{pred}|X_\text{train})$ then you are both adding noise to your estimate, and reducing the uncertainty in the estimate of $\theta$ (so you would probably be both overconfident and more wrong!), whereas if you optimise $P(X_\text{pred}|X_\text{train})$ I expect the main effect would be reducing the uncertainty in the posterior and thereby making your predictions overly confident (i.e. overfitting).

It's not bad and actually may improve quality quite a lot if you have a lot of unlabeled data (i.e. $X$'s without $Y$'s) and some labeled data: you train a model with labeled data, label unlabeled with your model and retrain the model. This falls into the class of so-called semi-supervised learning methods: http://en.wikipedia.org/wiki/Semi-supervised_learning

• Why would you do this if you have labels for the data? My understanding is that the label propagation you're referring to is only useful when you don't have labels associated with those observations. Nov 30, 2013 at 6:23
• yes, that's what I said "may improve quality quite a lot if you have a lot of unlabeled data (i.e. X's without Y's) and some labeled data". that's the situation when retraining model on it's own predictions may improve the model. Nov 30, 2013 at 14:08