0
$\begingroup$

I am trying to understand the concept of asymptotic unbiasedness. I understand that an estimator is said to be asymptotically unbiased if, when the size of our data increases to infinity, the bias of the estimator approaches 0.

However, this seems to conflict with what I have learned about bias in Machine Learning models. I have learned that increasing the number of examples used in training a machine learned model to infinity will not improve its prediction bias (see page three: https://www.cs.cmu.edu/~tom/10601_fall2012/exams/midterm_solutions.pdf)

However, the concept of asymptotic unbiasedness seems to be in conflict with this, since it seems to imply that increasing the amount of data can affect bias. Am I understanding this correctly?

$\endgroup$
4
  • $\begingroup$ A model trained on that data will have the same bias as what? $\endgroup$
    – Dave
    Commented Jan 7, 2021 at 20:15
  • $\begingroup$ As itself. This is referring to the model's prediction bias, and how it behaves when more training examples are used over time (edited the question for clarity). $\endgroup$ Commented Jan 7, 2021 at 20:16
  • $\begingroup$ It sounds like you want the machine learning model to be some kind of asymptotically unbiased estimator of something. Is this correct? $\endgroup$
    – Dave
    Commented Jan 7, 2021 at 20:18
  • $\begingroup$ It's not that I want the machine learned model to be asymptotically unbiased - it's more that (1) I have been told that bias in machine learning models is a function of model architecture, not the dataset size; (2) however, it seems like the bias of an estimator can vary based on the dataset size. I think it seems reasonable to look at a machine learned model as either a point estimator or a function estimator (see section on function estimation on page 122: deeplearningbook.org/contents/ml.html#pf21) $\endgroup$ Commented Jan 7, 2021 at 21:01

1 Answer 1

0
$\begingroup$

I think you misunderstand the asymptotic unbiasedness. Consider the following estimator of the variance: $\tilde S^2_n=\frac 1 n \sum_{i=1}^n(x_i-\bar x)^2$ as opposed to $S^2_n=\frac 1 {n-1} \sum_{i=1}^n(x_i-\bar x)^2$.

While $S^2_n$ is unbiased its brother $\tilde S^2_n$ is biased but asymptotically unbiased , because when $n\to\infty$ we have $\frac 1 n=\frac 1 {n-1}$.

What your ML lecture is talking about is a completely different thing. It's saying if you have an estimator like $S^2_n$ then increasing the sample size beyond some point stops being useful. In fact it also applies to $\tilde S^2_n$ because when $n$ is large enough you won't notice the diff with its unbiased better brother

$\endgroup$
5
  • $\begingroup$ Thanks for the answer! Quick clarification, I'm assuming you meant to say "While $\tilde S^2_n$ is unbiased its brother $S^2_n$ is biased but asymptotically unbiased?" $\endgroup$ Commented Jan 7, 2021 at 21:24
  • $\begingroup$ @user2621707, nope, the one with $n-1$ is unbiased, see e.g. en.wikipedia.org/wiki/Bias_of_an_estimator#Sample_variance $\endgroup$
    – Aksakal
    Commented Jan 7, 2021 at 21:33
  • $\begingroup$ Thanks! That helps a lot! Hmm, wouldn't your final statement "In fact it also applies to 𝑆̃ 2𝑛 because when 𝑛 is large enough you won't notice the diff with its unbiased better brother" actually prove that increasing the sample size helps reduce bias, since with more data the difference between the two estimators is negligible? $\endgroup$ Commented Jan 7, 2021 at 22:43
  • $\begingroup$ @user2621707, what I mean was that increasing sample size clearly helps, but not to infinity. At some large enough $n$ the benefit for the bias all but disappears. Your professor put "infinity" in the question, I doubt that he'd argue that larger sample is not better than smaller at any $n$. In practical settings $n>100$ is already large, because in most problems you're lucky to get 1% precision, truly. $\endgroup$
    – Aksakal
    Commented Jan 7, 2021 at 22:55
  • $\begingroup$ I see, that makes sense, thank you! $\endgroup$ Commented Jan 7, 2021 at 23:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.