49 votes
Accepted

What is a Highest Density Region (HDR)?

I recommend Rob Hyndman's 1996 article "Computing and Graphing Highest Density Regions" in The American Statistician. Here is the definition of the HDR, taken from that article: Let $f(x)$ be the ...
user avatar
45 votes
Accepted

Statistical tests when sample size is 1

Unfortunately, your student has a problem. The idea of any (inferential) statistical analysis is to understand whether a pattern of observations can be simply due to natural variation or chance, or ...
user avatar
34 votes

Maximum likelihood method vs. least squares method

I'd like to provide a straightforward answer. What is the main difference between maximum likelihood estimation (MLE) vs. least squares estimation (LSE) ? As @TrynnaDoStat commented, minimizing ...
user avatar
  • 5,366
34 votes
Accepted

Are there parameters where a biased estimator is considered "better" than the unbiased estimator?

One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are ...
user avatar
33 votes
Accepted

Why is the James-Stein estimator called a "shrinkage" estimator?

A picture is sometimes worth a thousand words, so let me share one with you. Below you can see an illustration that comes from Bradley Efron's (1977) paper Stein's paradox in statistics. As you can ...
user avatar
  • 117k
29 votes

Why is sample standard deviation a biased estimator of $\sigma$?

Complementing NRH's answer, if someone is teaching this to a group of students who haven't studied Jensen's inequality yet, one way to go is to define the sample standard deviation $$ S_n = \sqrt{\...
user avatar
  • 22.2k
29 votes
Accepted

Mean of the bootstrap sample vs statistic of the sample

Let's generalize, so as to focus on the crux of the matter. I will spell out the tiniest details so as to leave no doubts. The analysis requires only the following: The arithmetic mean of a set of ...
user avatar
  • 293k
29 votes
Accepted

Why would parametric statistics ever be preferred over nonparametric?

Rarely if ever a parametric test and a non-parametric test actually have the same null. The parametric $t$-test is testing the mean of the distribution, assuming the first two moments exist. The ...
user avatar
  • 29.4k
29 votes

Statistical tests when sample size is 1

Two-way ANOVA with One Observation per Cell After you finish your important 'lecture' about consulting a statistician before starting to take data, you can tell your student that there is barely ...
user avatar
  • 51.1k
28 votes
Accepted

What regression/estimation is not a MLE?

Least squares is indeed maximum likelihood if the errors are iid normal, but if they aren't iid normal, least squares is not maximum likelihood. For example if my errors were logistic, least squares ...
user avatar
  • 262k
27 votes

MLE vs MAP estimation, when to use which?

If a prior probability is given as part of the problem setup, then use that information (i.e. use MAP). If no such prior information is given or assumed, then MAP is not possible, and MLE is a ...
user avatar
  • 416
26 votes
Accepted

Do we ever use maximum likelihood estimation?

I am wondering if maximum likelihood estimation ever used in statistics. Certainly! Actually quite a lot -- but not always. We learn the concept of it but I wonder when it is actually used. When ...
user avatar
  • 262k
25 votes

What is "Targeted Maximum Likelihood Expectation"?

I agree that van der Laan has a tendency to invent new names for already existing ideas (e.g. the super-learner), but TMLE is not one of them as far as I know. It is actually a very clever idea, and I ...
user avatar
  • 8,032
25 votes

Why would parametric statistics ever be preferred over nonparametric?

As others have written: if the preconditions are met, your parametric test will be more powerful than the nonparametric one. In your watch analogy, the non-water-resistant one would be far more ...
user avatar
25 votes
Accepted

Is p-value a point estimate?

Point estimates and confidence intervals are for parameters that describe the distribution, e.g. mean or standard deviation. But unlike other sample statistics like the sample mean and the sample ...
user avatar
  • 7,009
25 votes

If a credible interval has a flat prior, is a 95% confidence interval equal to a 95% credible interval?

Many frequentist confidence intervals (CIs) are based on the likelihood function. If the prior distribution is truly non-informative, then the a Bayesian posterior has essentially the same information ...
user avatar
  • 51.1k
25 votes

Why are hypothesis tests still used when we have the bootstrap and central limit theorem?

Hypothesis tests are still used because they are motivated by a different need in statistical inference than interval estimators are motivated by. The purpose of a hypothesis test is to make a ...
user avatar
  • 26.7k
24 votes

An example of a consistent and biased estimator?

The simplest example I can think of is the sample variance that comes intuitively to most of us, namely the sum of squared deviations divided by $n$ instead of $n-1$: $$S_n^2 = \frac{1}{n} \sum_{i=1}^...
user avatar
  • 18.7k
24 votes

Is p-value a point estimate?

Yes, it could be (and has been) argued that a p-value is a point estimate. In order to identify whatever property of a distribution a p-value might estimate, we would have to assume it is ...
user avatar
  • 293k
23 votes
Accepted

Why do we need an estimator to be consistent?

If the estimator is not consistent, it won't converge to the true value in probability. In other words, there is always a probability that your estimator and true value will have a difference, no ...
user avatar
  • 52.5k
23 votes

Why do we need an estimator to be consistent?

Consider $n = 10\,000$ observations from the standard Cauchy distribution, which is the same as Student's t distribution with 1 degree of freedom. The tails of this distribution are sufficiently heavy ...
user avatar
  • 51.1k
22 votes
Accepted

Bias of moment estimator of lognormal distribution

There is something puzzling in those results since the first method provides an unbiased estimator of $\mathbb{E}[X^2]$, namely$$\frac{1}{N}\sum_{i=1}^N X_i^2$$has $\mathbb{E}[X^2]$ as its mean. ...
user avatar
  • 93.4k
22 votes

Statistical tests when sample size is 1

BruceET has described the proper analysis (Two-way ANOVA without interaction), so I'll put a more positive spin on the experiment. I'm assuming that the design was three pairs, where there is ...
user avatar
  • 2,249
21 votes

Are there parameters where a biased estimator is considered "better" than the unbiased estimator?

Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization). As for a mathematical ...
user avatar
18 votes
Accepted

Why $\sqrt{n}$ in the definition of asymptotic normality?

We don't get to choose here. The "normalizing" factor, in essence is a "variance-stabilizing to something finite" factor, so as for the expression not to go to zero or to infinity as sample size goes ...
user avatar
18 votes
Accepted

A description of the mean of the Geometric Distribution - is it unorthodox or just incorrect?

$\exp(\mathbb E[\log(X)])$ is the geometric mean of a positive random variable $X$ not the mean of a geometric random variable. So either the homework directions put the words in the wrong order, or ...
user avatar
  • 31.8k
17 votes
Accepted

Are estimates of regression coefficients uncorrelated?

This is an important consideration in designing experiments, where it can be desirable to have no (or very little) correlation among the estimates $\hat a$ and $\hat b$. Such lack of correlation can ...
user avatar
  • 293k
17 votes
Accepted

Growing number of Gaussians in a mixture

If your goal is to find the maximum-likelihood mixture of size $n+1$, then you can use the existing solution as an initialization, once you have enlarged it to have one more Gaussian. To enlarge it, ...
user avatar
  • 6,650
17 votes

Why do we need Bootstrapping?

Two answers. What's the standard error of the ratio of two means? What's the standard error of the median? What's the standard error of any complex statistic? Maybe there's a closed form equation, ...
user avatar
  • 14.9k

Only top scored, non community-wiki answers of a minimum length are eligible