How to understand randomness in the training data from bias-variance tradeoff?

The bias-variance decomposition of MSE is $MSE = (\mathbb{E}(\hat{\theta})-\theta)^2 + Var[\hat{\theta}]$. $\mathbb{E}(\hat{\theta})$ can be explained as the average from the randomness in the training data. In a real-world scenario, we are trying to build models based on a particular dataset, in this sense, what does it mean by "the randomness in the training data"? Is it relevant to sampling from the data?

Edit:

Suppose given a dataset D, it is easy to get an estimate of the parameter $\hat\theta$. However, $\mathbb{E}(\hat{\theta})$ implies there are multiple estimations of $\hat\theta$. My question is, why there are multiple estimations for the same dataset?

• If you're a Bayesian it is far less relevant. Jun 6 '16 at 23:18
• why is that? @probabilityislogic Jun 7 '16 at 0:04
• Are you quoting something directly when you say, "can be explained as the..."? If so, it'd be more helpful if you quoted it directly. However, with that said, it sounds like your resource might be a little awkwardly worded. Basically, on the right side of the equation the first term is the squared bias of the estimator and the second term is the variance of the estimator. As far as the bias goes, you're taking the difference between the expected value of the estimated parameter and the parameter itself. Is this making sense? Jun 7 '16 at 2:08
• If you're still confused then you might want to try re-wording your question. Jun 7 '16 at 2:19

That said, it's not quite true that you only ever get your hands on one estimate $$\hat \theta$$. Think about the process of cross validation. In cross validation we are essentially simulating the situation where we have sampled multiple concrete training data sets from the same population, we fit our model once to each such training data set, and we receive multiple estimations $$\hat \theta$$. We then often take the mean of the losses of these estimators on the test set, which is used to tune parameters or estimate the out of sample loss.