In Statistics, High Variance represents that:

Overall, elements of dataset are widely dispersed/different/spread out from the mean and from each other.

In Machine Learning, it represents:

Our parameters have over-fitted on the dataset and so we will get unnecessary curves and will not be able to predict on unseen data.

How these two concepts can be compared? Or are these independent concepts with no direct relation?

  • 1
    $\begingroup$ Not quite. Both uses of high variances are statistical. Since part of machine learning is built on a statistical foundation (such as the part that talks about variance), it might have inherited one or both of these. $\endgroup$ – Richard Hardy Mar 11 '18 at 8:05

In statistics, variance is not a property of a (sample) dataset. It is a property of random variables. (You can calculate sample variance for an observed sample from a random variable. This is what you are referring to, but it is a secondary concept.)

As such, we can talk of the variance of a parameter estimate, e.g., of a regression parameter. The idea is that we can draw many samples from the underlying data population, each sample of size $n$, and calculate the regression parameter estimate $\hat{\beta}$ each time. Then all these calculated $\hat{\beta}$s may have a high or a low variance. Lower variance for estimates is typically better.

And there is your connection to overfitting. If you overfit your model, your parameter estimates will have a high variance and depend very much on the specific data you have. So the model will not be able to predict well. The concept of overfitting certainly also exists in "non-ML" statistics, and predates the usage in ML.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.