I have read about what a high variance and high bias model is and everywhere the emphasis is more on the consequences of either. I am confused as to what the random variable is when we are talking about a model having high variance or high bias and how can i get different realizations of this random variable?
In the context of parameter estimation (where the expected squared estimation error is additively decomposed into variance and squared bias), the random variable would be the vector of derived parameter estimators that best characterize the true data generating process (DGP) in terms of the parameter estimators of the model (which need not correspond to the true DGP).
Note that it would not be the vector of parameter estimators for the so called "pseudo-true" model parameters defining the best population-level approximation of the DGP; such an interpretation would ignore the model bias (the difference between the functional form of the true DGP and the model).
For example, if the DGP is
y=\beta_0+\beta_1 x_1+\beta_2 x_2+u
while the model is
the random variable would be an estimator of $(\beta_0,\beta_1,\beta_2)^\top$ expressed in terms of $(\hat\gamma_0,\hat\gamma_1)^\top$.
(It would not be simply $(\hat\gamma_0,\hat\gamma_1)^\top$.)
In the context of prediction (where the expected squared prediction error is additively decomposed into variance, squared bias and irreducible error), the random variable would be the squared prediction error.*
*This could be generalized to some other figure of merit (using the words of @cbeleites) in place of squared prediction error, but the decomposition is algebraically neatest for squared prediction error.
The high variance and/or high bias relates to a "statistic" you are using to evaluate your model compared to the ground truth. This statistic is generally a random variable (accuracy, precision, recall) or a posterior mean, median, mode.
The statistic, lets say accuracy, is determined by the data and parameters of the model. $$p(acc | X, y,\theta)$$ I see this as the random variable determining number of events predicted correctly (using the measure of your statistic - mean square error, mean absolute error, log loss, hinge loss or whatever you fancy)
Edit: if you have no ground truth like clustering, then the concept of high bias cannot occur
The random variable is not the mean square error of the prediction or the measure of model performance, the random variable is the model itself.
Since the training data is random (at least the outcome $y$ is), the model will be random too, and you can see this testing your model on a separate (test) set.
Of course statistical models don't seem to be "random variables", but in some sense they are, you can see this in two ways at least:
- Prediction models can be seen as functions of predictors $f(x)$. This predicted outcome is random, and its MSE is the sum of its variance and its squared bias, as central to the famous trade-off argument. Predictions of the model are the random variable, but the model itself is a function that makes predictions.
- If you are more concerned about inference, you may prefer to see your model as an equation that uses some parameters. Those parameters are random, and to minimise their MSE you may turn to trade-off argument, as before. Of course predictions depend on parameters, so reducing MSE of parameters reduces MSE of $f(x)$ too.