45

Unfortunately, your student has a problem. The idea of any (inferential) statistical analysis is to understand whether a pattern of observations can be simply due to natural variation or chance, or whether there is something systematic there. If the natural variation is large, then the observed difference may be simply due to chance. If the natural ...


43

I recommend Rob Hyndman's 1996 article "Computing and Graphing Highest Density Regions" in The American Statistician. Here is the definition of the HDR, taken from that article: Let $f(x)$ be the density function of a random variable $X$. Then the $100(1-\alpha)\%$ HDR is the subset $R(f_\alpha)$ of the sample space of $X$ such that $$R(f_\alpha) = ...


37

This may be considered... cheating, but the OLS estimator is a MoM estimator. Consider a standard linear regression specification (with $K$ stochastic regressors, so magnitudes are conditional on the regressor matrix), and a sample of size $n$. Denote $s^2$ the OLS estimator of the variance $\sigma^2$ of the error term. It is unbiased so $$ MSE(s^2) = \...


34

One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are biased but have much lower variance. E.g. install.packages("ridge") library(ridge) set.seed(831) data(GenCont) ridgemod <- linearRidge(Phenotypes ~ ., data = ...


32

I'd like to provide a straightforward answer. What is the main difference between maximum likelihood estimation (MLE) vs. least squares estimation (LSE) ? As @TrynnaDoStat commented, minimizing squared error is equivalent to maximizing the likelihood in this case. As said in Wikipedia, In a linear model, if the errors belong to a normal distribution the ...


32

A picture is sometimes worth a thousand words, so let me share one with you. Below you can see an illustration that comes from Bradley Efron's (1977) paper Stein's paradox in statistics. As you can see, what Stein's estimator does is move each of the values closer to the grand average. It makes values greater than the grand average smaller, and values ...


29

Let's generalize, so as to focus on the crux of the matter. I will spell out the tiniest details so as to leave no doubts. The analysis requires only the following: The arithmetic mean of a set of numbers $z_1, \ldots, z_m$ is defined to be $$\frac{1}{m}\left(z_1 + \cdots + z_m\right).$$ Expectation is a linear operator. That is, when $Z_i, i=1,\ldots,...


29

Rarely if ever a parametric test and a non-parametric test actually have the same null. The parametric $t$-test is testing the mean of the distribution, assuming the first two moments exist. The Wilcoxon rank sum test does not assume any moments, and tests equality of distributions instead. Its implied parameter is a weird functional of distributions, the ...


28

Complementing NRH's answer, if someone is teaching this to a group of students who haven't studied Jensen's inequality yet, one way to go is to define the sample standard deviation $$ S_n = \sqrt{\sum_{i=1}^n\frac{(X_i-\bar{X}_n)^2}{n-1}} , $$ suppose that $S_n$ is non degenerate (therefore, $\mathrm{Var}[S_n]\ne0$), and notice the equivalences $$ 0 <...


28

Least squares is indeed maximum likelihood if the errors are iid normal, but if they aren't iid normal, least squares is not maximum likelihood. For example if my errors were logistic, least squares wouldn't be a terrible idea but it wouldn't be maximum likelihood. Lots of estimators are not maximum likelihood estimators; while maximum likelihood estimators ...


28

Two-way ANOVA with One Observation per Cell After you finish your important 'lecture' about consulting a statistician before starting to take data, you can tell your student that there is barely enough data here to support a legitimate experimental design. If the subjects were chosen at random from some relevant population, glucose determinations were ...


26

If a prior probability is given as part of the problem setup, then use that information (i.e. use MAP). If no such prior information is given or assumed, then MAP is not possible, and MLE is a reasonable approach.


26

I am wondering if maximum likelihood estimation ever used in statistics. Certainly! Actually quite a lot -- but not always. We learn the concept of it but I wonder when it is actually used. When people have a parametric distributional model, they quite often choose to use maximum likelihood estimation. When the model is correct, there are a number of ...


25

There are a few potential ways for you to keep the gender dummy in a fixed effects regression. Within Estimator Suppose you have a similar model compared to your pooled OLS model which is $$y_{it} = \beta_1 + \sum^{10}_{t=2} \beta_t d_t + \gamma_1 (male_i) + \sum^{10}_{t=1} \gamma_t (d_t \cdot male_i) + X'_{it}\theta + c_i + \epsilon_{it}$$ where the ...


25

I agree that van der Laan has a tendency to invent new names for already existing ideas (e.g. the super-learner), but TMLE is not one of them as far as I know. It is actually a very clever idea, and I have seen nothing from the Machine Learning community which looks similar (although I might just be ignorant). The ideas come from the theory of semiparametric-...


25

As others have written: if the preconditions are met, your parametric test will be more powerful than the nonparametric one. In your watch analogy, the non-water-resistant one would be far more accurate unless it got wet. For instance, your water-resistant watch might be off by one hour either way, whereas the non-water-resistant one would be accurate... ...


25

Many frequentist confidence intervals (CIs) are based on the likelihood function. If the prior distribution is truly non-informative, then the a Bayesian posterior has essentially the same information as the likelihood function. Consequently, in practice, a Bayesian probability interval (or credible interval) may be very similar numerically to a frequentist ...


25

Hypothesis tests are still used because they are motivated by a different need in statistical inference than interval estimators are motivated by. The purpose of a hypothesis test is to make a decision as to whether there is evidence for the alternative hypothesis' expression of the population parameter. Confidence intervals serve a different purpose: they ...


24

The simplest example I can think of is the sample variance that comes intuitively to most of us, namely the sum of squared deviations divided by $n$ instead of $n-1$: $$S_n^2 = \frac{1}{n} \sum_{i=1}^n \left(X_i-\bar{X} \right)^2$$ It is easy to show that $E\left(S_n^2 \right)=\frac{n-1}{n} \sigma^2$ and so the estimator is biased. But assuming finite ...


24

Point estimates and confidence intervals are for parameters that describe the distribution, e.g. mean or standard deviation. But unlike other sample statistics like the sample mean and the sample standard deviation the p-value is not an useful estimator of an interesting distribution parameter. Look at the answer by @whuber for technical details. The p-...


23

As stated in comment, the prior distribution represents prior beliefs about the distribution of the parameters. When prior beliefs are actually available, you can: convert them in terms of moments (e.g. mean and variance) to fit a common distribution to these moments (e.g. Gaussian if your parameter lies to the real line, Gamma if it lies to $R^+$). use ...


23

Yes, it could be (and has been) argued that a p-value is a point estimate. In order to identify whatever property of a distribution a p-value might estimate, we would have to assume it is asymptotically unbiased. But, asymptotically, the mean p-value for the null hypothesis is $1/2$ (ideally; for some tests it might be some other nonzero number) and for any ...


23

If the estimator is not consistent, it won't converge to the true value in probability. In other words, there is always a probability that your estimator and true value will have a difference, no matter how many data points you have. This is actually bad, because even if you collect immense amount of data, your estimate will always have a positive ...


22

There is something puzzling in those results since the first method provides an unbiased estimator of $\mathbb{E}[X^2]$, namely$$\frac{1}{N}\sum_{i=1}^N X_i^2$$has $\mathbb{E}[X^2]$ as its mean. Hence the blue dots should be around the expected value (orange curve); the second method provides a biased estimator of $\mathbb{E}[X^2]$, namely$$\mathbb{E}[\exp(...


22

Consider $n = 10\,000$ observations from the standard Cauchy distribution, which is the same as Student's t distribution with 1 degree of freedom. The tails of this distribution are sufficiently heavy that it has no mean; the distribution is centered at its median $\eta = 0.$ A sequence of sample means $A_j = \frac 1j \sum_{i=1}^j X_i$ is not consistent for ...


22

BruceET has described the proper analysis (Two-way ANOVA without interaction), so I'll put a more positive spin on the experiment. I'm assuming that the design was three pairs, where there is variability between pairs. One of each pair was given insulin and the other without, hopefully randomized. Then each sample (pair X treatment, I call the ...


21

Estimators are statistics, and statistics have sampling distributions (that is, we're talking about the situation where you keep drawing samples of the same size and looking at the distribution of the estimates you get, one for each sample). The quote is referring to the distribution of MLEs as sample sizes approach infinity. So let's consider an explicit ...


21

Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization). As for a mathematical example, I am pulling the following example from the excellent Statistical Inference by Casella and Berger to show that a biased estimator has lower Mean Squared ...


20

Consider a multinomial logit model in which you estimate market shares as $$\widehat{s}_{jt} = \frac{\exp(\delta_{jt})}{1 + \sum^{J}_{g=1}\exp(\delta_{gt})}$$ where the outside good is normalized to zero. When you take the log of this expression, you get $$\log (\widehat{s}_{jt}) = \delta_{jt} – \log \left( 1 + \sum^{J}_{g=1}\exp(\delta_{gt}) \right)$$ for ...


Only top voted, non community-wiki answers of a minimum length are eligible