0
$\begingroup$

I am trying to understand the concept of asymptotic unbiasedness. I understand that an estimator is said to be asymptotically unbiased if, when the size of our data increases to infinity, the bias of the estimator approaches 0.

However, this seems to conflict with what I have learned about bias in Machine Learning models. I have learned that increasing the number of examples used in training a machine learned model to infinity will not improve its prediction bias (see page three: https://www.cs.cmu.edu/~tom/10601_fall2012/exams/midterm_solutions.pdf)

However, the concept of asymptotic unbiasedness seems to be in conflict with this, since it seems to imply that increasing the amount of data can affect bias. Am I understanding this correctly?

Improve this question
$\endgroup$
4
  • $\begingroup$ A model trained on that data will have the same bias as what? $\endgroup$
    – Dave
    Jan 7, 2021 at 20:15
  • $\begingroup$ As itself. This is referring to the model's prediction bias, and how it behaves when more training examples are used over time (edited the question for clarity). $\endgroup$
    – user2621707
    Jan 7, 2021 at 20:16
  • $\begingroup$ It sounds like you want the machine learning model to be some kind of asymptotically unbiased estimator of something. Is this correct? $\endgroup$
    – Dave
    Jan 7, 2021 at 20:18
  • $\begingroup$ It's not that I want the machine learned model to be asymptotically unbiased - it's more that (1) I have been told that bias in machine learning models is a function of model architecture, not the dataset size; (2) however, it seems like the bias of an estimator can vary based on the dataset size. I think it seems reasonable to look at a machine learned model as either a point estimator or a function estimator (see section on function estimation on page 122: deeplearningbook.org/contents/ml.html#pf21) $\endgroup$
    – user2621707
    Jan 7, 2021 at 21:01

1 Answer 1

Reset to default
0
$\begingroup$

I think you misunderstand the asymptotic unbiasedness. Consider the following estimator of the variance: $\tilde S^2_n=\frac 1 n \sum_{i=1}^n(x_i-\bar x)^2$ as opposed to $S^2_n=\frac 1 {n-1} \sum_{i=1}^n(x_i-\bar x)^2$.

While $S^2_n$ is unbiased its brother $\tilde S^2_n$ is biased but asymptotically unbiased , because when $n\to\infty$ we have $\frac 1 n=\frac 1 {n-1}$.

What your ML lecture is talking about is a completely different thing. It's saying if you have an estimator like $S^2_n$ then increasing the sample size beyond some point stops being useful. In fact it also applies to $\tilde S^2_n$ because when $n$ is large enough you won't notice the diff with its unbiased better brother

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the answer! Quick clarification, I'm assuming you meant to say "While $\tilde S^2_n$ is unbiased its brother $S^2_n$ is biased but asymptotically unbiased?" $\endgroup$
    – user2621707
    Jan 7, 2021 at 21:24
  • $\begingroup$ @user2621707, nope, the one with $n-1$ is unbiased, see e.g. en.wikipedia.org/wiki/Bias_of_an_estimator#Sample_variance $\endgroup$
    – Aksakal
    Jan 7, 2021 at 21:33
  • $\begingroup$ Thanks! That helps a lot! Hmm, wouldn't your final statement "In fact it also applies to 𝑆̃ 2𝑛 because when 𝑛 is large enough you won't notice the diff with its unbiased better brother" actually prove that increasing the sample size helps reduce bias, since with more data the difference between the two estimators is negligible? $\endgroup$
    – user2621707
    Jan 7, 2021 at 22:43
  • $\begingroup$ @user2621707, what I mean was that increasing sample size clearly helps, but not to infinity. At some large enough $n$ the benefit for the bias all but disappears. Your professor put "infinity" in the question, I doubt that he'd argue that larger sample is not better than smaller at any $n$. In practical settings $n>100$ is already large, because in most problems you're lucky to get 1% precision, truly. $\endgroup$
    – Aksakal
    Jan 7, 2021 at 22:55
  • $\begingroup$ I see, that makes sense, thank you! $\endgroup$
    – user2621707
    Jan 7, 2021 at 23:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.