65

Yes. Often it is the case that we are interested in minimizing the mean squared error, which can be decomposed into variance + bias squared. This is an extremely fundamental idea in machine learning, and statistics in general. Frequently we see that a small increase in bias can come with a large enough reduction in variance that the overall MSE decreases. A ...


61

Most proofs I have seen are simple enough that Gauss (however he did it) probably found it pretty easy to prove. I've been looking for a derivation on CV that I could link you to (there are a number of links to proofs off-site, including at least one in answers here), but I haven't found one here on CV in a couple of searches, so for the sake of ...


42

The correction is called Bessel's correction and it has a mathematical proof. Personally, I was taught it the easy way: using $n-1$ is how you correct the bias of $E[\frac{1}{n}\sum_1^n(x_i - \bar x)^2]$ (see here). You can also explain the correction based on the concept of degrees of freedom, simulation isn't strictly needed.


42

If the population is known to be normal, a 95% confidence interval based on a single observation $x$ is given by $$x \pm 9.68 \left| x \right| $$ This is discussed in the article "An Effective Confidence Interval for the Mean With Samples of Size One and Two," by Wall, Boen, and Tweedie, The American Statistician, May 2001, Vol. 55, No.2. (pdf)


39

Importance sampling is a simulation or Monte Carlo method intended for approximating integrals. The term "sampling" is somewhat confusing in that it does not intend to provide samples from a given distribution. The intuition behind importance sampling is that a well-defined integral, like $$\mathfrak{I}=\int_\mathfrak{X} h(x)\,\text{d}x$$ can be expressed ...


37

According to Weisstein's World of Mathematics, it was first proved by Gauss in 1823. The reference is volume 4 of Gauss' Werke, which can be read at https://archive.org/details/werkecarlf04gausrich. The relevant pages seem to be 47-49. It seems that Gauss investigated the question and came up with a proof. I don't read Latin, but there is a German summary in ...


34

For the more restricted question Why is a biased standard deviation formula typically used? the simple answer Because the associated variance estimator is unbiased. There is no real mathematical/statistical justification. may be accurate in many cases. However, this is not necessarily always the case. There are at least two important aspects of ...


34

One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are biased but have much lower variance. E.g. install.packages("ridge") library(ridge) set.seed(831) data(GenCont) ridgemod <- linearRidge(Phenotypes ~ ., data = ...


33

Covariances are variances. Since by the polarization identity $$\newcommand{\c}{\text{Cov}}\newcommand{\v}{\text{Var}} \c(X,Y) = \v\left(\frac{X+Y}{2}\right) - \v\left(\frac{X-Y}{2}\right),$$ the denominators must be the same.


33

I am a Bayesian, but I find these kinds of criticisms against "frequentists" to be overstated and unfair. Both Bayesians and classical statisticians accept all the same mathematical results to be true, so there is really no dispute here about the properties of the various estimators. Even if you are a Bayesian, it is clearly true that the sample mean is no ...


30

@cardinal gave a great answer (+1), but the whole issue remains mysterious unless one is familiar with the proofs (and I am not). So I think the question remains as to what is an intuitive reason that Stein's paradox does not appear in $\mathbb R$ and $\mathbb R^2$. I find very helpful a regression perspective offered in Stephen Stigler, 1990, A Galtonian ...


29

A few more steps of the Bias - Variance decomposition Indeed, the full derivation is rarely given in textbooks as it involves a lot of uninspiring algebra. Here is a more complete derivation using notation from the book "Elements of Statistical Learning" on page 223 If we assume that $Y = f(X) + \epsilon$ and $E[\epsilon] = 0$ and $Var(\epsilon) = \sigma^...


28

Sure there is. Use a Bayesian paradigm. Chances are you have at least some idea of what $\mu$ could possibly be - for instance, that it physically cannot be negative, or that it obviously cannot be larger than 100 (maybe you are measuring the height of your local high school football team members in feet). Put a prior on that, update it with your lone ...


28

I will try to add to the other answer. First, completeness is a technical condition which is justified mainly by the theorems that use it. So let us start with some related concepts and theorems where they occur. Let $X=(X_1,X_2,\dotsc,X_n)$ represent a vector of iid data, which we model as having a distribution $f(x;\theta), \theta \in \Theta$ where the ...


28

You are not wrong, but you made an error in one step since $E[(f(x)-f_k(x))^2] \ne Var(f_k(x))$. $E[(f(x)-f_k(x))^2]$ is $\text{MSE}(f_k(x)) = Var(f_k(x)) + \text{Bias}^2(f_k(x))$. \begin{align*} E[(Y-f_k(x))^2]& = E[(f(x)+\epsilon-f_k(x))^2] \\ &= E[(f(x)-f_k(x))^2]+2E[(f(x)-f_k(x))\epsilon]+E[\epsilon^2]\\ &= E\left[\left(f(x) - E(f_k(x)) + E(...


28

In short, regularization changes the distribution of the test statistic, rendering tests of hypothesis moot. In instances where we want to use regression to make inferences about interventions, we want unbiasedness. Not everything to do with data is a prediction problem.


21

They are related ideas, but an asymptotically unbiased estimator doesn't have to be consistent. For example, imagine an i.i.d. sample of size $n$ ($X_1, X_2, ..., X_n$) from some distribution with mean $\mu$ and variance $\sigma^2$. As an estimator of $\mu$ consider $T = X_1 + 1/n$. (Edit: Note the $X_1$ there, not $\bar{X}$) The bias is $1/n$ so $T$ is ...


21

Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization). As for a mathematical example, I am pulling the following example from the excellent Statistical Inference by Casella and Berger to show that a biased estimator has lower Mean Squared ...


20

A special case ought to give you an intuition; think about the following: $$\hat{\mathbb{Cov}}\left(X, X\right)= \hat{\mathbb{V}}\left(X\right)$$ You are happy that the latter is $\frac{\sum_{i=1}^{n}\left(X_{i}-\overline{X}\right)^{2}}{n-1}$ due to the Bessel correction. But replacing $Y$ by $X$ in $\hat{\mathbb{Cov}}\left(X, Y\right)$ for the the former ...


19

One example that comes to mind is some GLS estimator that weights observations differently although that is not necessary when the Gauss-Markov assumptions are met (which the statistician may not know to be the case and hence apply still apply GLS). Consider the case of a regression of $y_i$, $i=1,\ldots,n$ on a constant for illustration (readily ...


19

But generally, if we have an unbiased MLE, would it also be the best unbiased estimator ? If there is a complete sufficient statistics, yes. Proof: Lehmann–Scheffé theorem: Any unbiased estimator that is a function of a complete sufficient statistics is the best (UMVUE). MLE is a function of any sufficient statistics. See 4.2.3 here; Thus an unbiased ...


19

Generally, refitting using no penalty after having done variable selection via the Lasso is considered "cheating" since you have already looked at the data and the resulting p-values and confidence intervals are not valid in the usual sense. This very recent paper looks at exactly what you want to do, and explains conditions under which fitting a lasso, ...


18

Consider the simple binary logistic regression model, with a binary dependent variable and only a constant and a binary regressor $T$. $$\Pr(Y_i=1\mid T_i=1) = \Lambda (\alpha + \beta T_i)$$ where $\Lambda$ is the logistic cdf, $\Lambda(u) = \left[1+\exp\{-u\}\right]^{-1}$. In logit form we have $$\ln \left(\frac{\Pr(Y_i=1\mid T_i=1)}{1-\Pr(Y_i=1\mid T_i=1)}...


17

Importance sampling is a form of sampling from a distribution different from the distribution of interest so as to more easily obtain better estimates of a parameter from the distribution of interest. Typically this will provide estimates of the parameter with a lower variance than would be obtained by sampling directly from the original distribution with ...


16

1) No it isn't. 2) because the calculation of the distribution of the test statistic relies on using the square root of the ordinary Bessel-corrected variance to get the estimate of standard deviation. If it were included it would only scale each t-statistic - and hence its distribution - by a factor (a different one at each d.f.); that would then scale ...


16

To add to the previous responses. You should definitely check out the recent work by Tibshirani and colleagues. They have developed a rigorous framework for inferring selection-corrected p-values and confidence intervals for lasso-type methods and also provide an R-package. See: Lee, Jason D., et al. "Exact post-selection inference, with application to the ...


15

In my opinion, the question is not truly coherent in that the maximisation of a likelihood and unbiasedness do not get along, if only because maximum likelihood estimators are equivariant, ie the transform of the estimator is the estimator of the transform of the parameter, while unbiasedness does not stand under non-linear transforms. Therefore, maximum ...


15

Technically what you're describing when you say that your estimator gets closer to the true value as the sample size grows is (as others have mentioned) consistency, or convergence of statistical estimators. This convergence can either be convergence in probability, which says that $\lim_{n \to \infty} P(|\hat{\theta}_n - \theta| > \epsilon) = 0$ for ...


15

The sample standard deviation $S=\sqrt{\frac{\sum (X - \bar{X})^2}{n-1}}$ is complete and sufficient for $\sigma$ so the set of unbiased estimators of $\sigma^k$ given by $$ \frac{(n-1)^\frac{k}{2}}{2^\frac{k}{2}} \cdot \frac{\Gamma\left(\frac{n-1}{2}\right)}{\Gamma\left(\frac{n+k-1}{2}\right)} \cdot S^k = \frac{S^k}{c_k} $$ (See Why is sample standard ...


Only top voted, non community-wiki answers of a minimum length are eligible