1
$\begingroup$

I have read about what a high variance and high bias model is and everywhere the emphasis is more on the consequences of either. I am confused as to what the random variable is when we are talking about a model having high variance or high bias and how can i get different realizations of this random variable?

$\endgroup$
3
$\begingroup$

In the context of parameter estimation (where the expected squared estimation error is additively decomposed into variance and squared bias), the random variable would be the vector of derived parameter estimators that best characterize the true data generating process (DGP) in terms of the parameter estimators of the model (which need not correspond to the true DGP).

Note that it would not be the vector of parameter estimators for the so called "pseudo-true" model parameters defining the best population-level approximation of the DGP; such an interpretation would ignore the model bias (the difference between the functional form of the true DGP and the model).

For example, if the DGP is $$ y=\beta_0+\beta_1 x_1+\beta_2 x_2+u $$ while the model is $$ y=\gamma_0+\gamma_1 x_1+v, $$ the random variable would be an estimator of $(\beta_0,\beta_1,\beta_2)^\top$ expressed in terms of $(\hat\gamma_0,\hat\gamma_1)^\top$.
(It would not be simply $(\hat\gamma_0,\hat\gamma_1)^\top$.)


In the context of prediction (where the expected squared prediction error is additively decomposed into variance, squared bias and irreducible error), the random variable would be the squared prediction error.*

*This could be generalized to some other figure of merit (using the words of @cbeleites) in place of squared prediction error, but the decomposition is algebraically neatest for squared prediction error.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ 1. did you mean prediction where you wrote expected squared prediction error. 2 How do you achieve different realizations of this prediction? Each time it is going to predict the same probability only as i see it? $\endgroup$ – MiloMinderbinder Oct 31 '19 at 15:30
  • 1
    $\begingroup$ @MiloMinderbinder, 1. I was wrong; now edited. 2. The prediction (as well as parameter estimates) depends on the sample that is drawn from the population. There will be a distribution of predictions (as well as parameter estimates) over the different possible samples. $\endgroup$ – Richard Hardy Oct 31 '19 at 15:56
  • $\begingroup$ @MiloMinderbinder, I have though more about the issue and updated my answer again. Still struggling a bit with a precise formulation. Yours is a challenging question. $\endgroup$ – Richard Hardy Nov 1 '19 at 7:57
  • 1
    $\begingroup$ @MiloMinderbinder: different realizations when looking at parameters (the model itself) can result from using independent but IID training data. Variance and bias of prediction figure of merit can result from such variance + bias in teh model and of the figure of merit estimation procedure and the test data (again look at intependent but IID test samples) $\endgroup$ – cbeleites unhappy with SX Nov 1 '19 at 19:11
  • 1
    $\begingroup$ @cbeleitessupportsMonica, edited to incorporate your suggestion (though not verbatim). I hope this is alright. $\endgroup$ – Richard Hardy Nov 1 '19 at 19:58
0
$\begingroup$

The high variance and/or high bias relates to a "statistic" you are using to evaluate your model compared to the ground truth. This statistic is generally a random variable (accuracy, precision, recall) or a posterior mean, median, mode.

The statistic, lets say accuracy, is determined by the data and parameters of the model. $$p(acc | X, y,\theta)$$ I see this as the random variable determining number of events predicted correctly (using the measure of your statistic - mean square error, mean absolute error, log loss, hinge loss or whatever you fancy)

Edit: if you have no ground truth like clustering, then the concept of high bias cannot occur

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

The random variable is not the mean square error of the prediction or the measure of model performance, the random variable is the model itself.

Since the training data is random (at least the outcome $y$ is), the model will be random too, and you can see this testing your model on a separate (test) set.

Of course statistical models don't seem to be "random variables", but in some sense they are, you can see this in two ways at least:

  • Prediction models can be seen as functions of predictors $f(x)$. This predicted outcome is random, and its MSE is the sum of its variance and its squared bias, as central to the famous trade-off argument. Predictions of the model are the random variable, but the model itself is a function that makes predictions.
  • If you are more concerned about inference, you may prefer to see your model as an equation that uses some parameters. Those parameters are random, and to minimise their MSE you may turn to trade-off argument, as before. Of course predictions depend on parameters, so reducing MSE of parameters reduces MSE of $f(x)$ too.
| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.