The bias-variance decomposition of MSE is $MSE = (\mathbb{E}(\hat{\theta})-\theta)^2 + Var[\hat{\theta}]$. $\mathbb{E}(\hat{\theta})$ can be explained as the average from the randomness in the training data. In a real-world scenario, we are trying to build models based on a particular dataset, in this sense, what does it mean by "the randomness in the training data"? Is it relevant to sampling from the data?


Suppose given a dataset D, it is easy to get an estimate of the parameter $\hat\theta$. However, $\mathbb{E}(\hat{\theta})$ implies there are multiple estimations of $\hat\theta$. My question is, why there are multiple estimations for the same dataset?

  • $\begingroup$ If you're a Bayesian it is far less relevant. $\endgroup$ Jun 6 '16 at 23:18
  • $\begingroup$ why is that? @probabilityislogic $\endgroup$ Jun 7 '16 at 0:04
  • $\begingroup$ Are you quoting something directly when you say, "can be explained as the..."? If so, it'd be more helpful if you quoted it directly. However, with that said, it sounds like your resource might be a little awkwardly worded. Basically, on the right side of the equation the first term is the squared bias of the estimator and the second term is the variance of the estimator. As far as the bias goes, you're taking the difference between the expected value of the estimated parameter and the parameter itself. Is this making sense? $\endgroup$
    – Steve S
    Jun 7 '16 at 2:08
  • $\begingroup$ If you're still confused then you might want to try re-wording your question. $\endgroup$
    – Steve S
    Jun 7 '16 at 2:19

The Bias-Variance tradeoff is a theoretical tool for thinking about machine learning, not generally a practical tool for doing machine learning. This is an important distinction, we use this principle to guide our thinking in different scenarios, and anticipate how our models may behave in certain situations.

To achieve this, we do conceive of our concrete training data set as one of a universe of possibilities. It's often very realistic to imagine that if you were to go through the process of collecting training data again, you would get a different particular data set, but it would still be informative of the same real world process. In this sense, while we only ever get one concrete training data set, it is one of many we could have received. We're not very interested in learning things about the particular training data set we happened to get, we're interested in learning things about the process that created that training data. So in that sense, we're much more interested in all of the training data sets we could have received, than the one we happened to recieve.

That said, it's not quite true that you only ever get your hands on one estimate $\hat \theta$. Think about the process of cross validation. In cross validation we are essentially simulating the situation where we have sampled multiple concrete training data sets from the same population, we fit our model once to each such training data set, and we receive multiple estimations $\hat \theta$. We then often take the mean of the losses of these estimators on the test set, which is used to tune parameters or estimate the out of sample loss.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.