# Sufficient Statistic and Maximum likelihood

This is more a conceptual question, but it seems to me that a sufficient statistic for a parameter is a concepts that applies only if we want to estimate the parameter via maximum likelihood. Is this right or wrong?

People usually claim that sufficient statistics are a data reduction technique. That is, I can store only the summary data expressed by the form of the observed value of the sufficient statistic and I will have enough information to make inference about the parameter of interest, $\theta$, in the future. To me, this is true only if we will make inference via maximum likelihood.

To expand, I might be able to come up with a better estimator for $\theta$ which takes on a strange functional form, and maybe then the observed value of the sufficient statistic will not be enough to allow me to infer about $\theta$ in the way that I wish.

• We ought first to establish that this question actually has some content. Could you provide us an example of any reasonable statistical procedure that requires more information than the likelihood?
– whuber
Jan 26, 2018 at 18:54
• well, if I think of the method of moments estimator, there is no guarantee that the resulting method of moments estimator will be a function of the sufficient statistic, is there? Jan 26, 2018 at 19:30
• You might want to look at the Rao-Blackwell Theorem en.wikipedia.org/wiki/Rao–Blackwell_theorem for more information about the usefulness of sufficient statistics. Jan 26, 2018 at 19:33
• Hmm interesting, so given an estimator g(x), obtained whichever way I want, then I can construct one that is a function of a sufficient statistic via E[g(x)|T(x)] and it will be no worse in terms of the expected square distance. This is nice, but now we have had to restrict what we mean be a "good estimator" which is nowhere considered when talking about sufficient statistics. Jan 26, 2018 at 19:45
• That's a fair criticism. It gets to the heart of the problem many people have with maximum likelihood in the first place: it doesn't account at all for any loss function. However, expected squared loss enjoys some rather general properties. Furthermore, anyone using MM estimators isn't considering a loss function either, so the same criticism can (and should be) be leveled at them.
– whuber
Jan 26, 2018 at 21:23

It is not true that sufficient statistics are only relevant with maximum likelihood estimators. For instance, knowing a sufficient statistic is enough for Bayesian inference, see the calculation in Reducing dimension $P(\theta|y) = P(\theta|s)$ in the posterior distribution.

People usually claim that sufficient statistics are a data reduction technique. That is, I can store only the summary data expressed by the form of the observed value of the sufficient statistic and I will have enough information to make inference about the parameter of interest, 𝜃, in the future. To me, this is true only if we will make inference via maximum likelihood.

See the Rao-Blackwell theorem: Understanding the Rao-Blackwell Theorem and at Wikipedia.

Some other relevant posts:

If there is a sufficient statistic, then it contains all the information that a sample has about the distribution.

Other statistics that describe the sample are superfluous.

To expand, I might be able to come up with a better estimator for $$\theta$$ which takes on a strange functional form, and maybe then the observed value of the sufficient statistic will not be enough to allow me to infer about $$\theta$$ in the way that I wish.

If any estimation method that requires the additional statistics would be better, then it is despite of it and not because of it. We could replace that estimation method by a method that is dependent on the sufficient statistic only and have the additional statistics randomly generated and independent from the actual sample.

### Example

The example is a simple case with a normal distribution. It gives a simple example of how the Fisher Neyman factorisation theorem represents a sample basically as composed of two independent groups of statistics, one that depend on the distribution parameters, and one that do not depend on the distribution parameters. The statistics from the latter group can be replaced with surrogate data without influencing the behaviour of the estimation method.

If we sample a distribution that follows a normal distribution $$X_i \sim N(\mu,1)$$ then we can replace a sample $$X_1, X_2, \dots, X_n$$ with surrogate data that depends on the original sample only via the sufficient statistic, the sample mean $$\bar{X}$$ $$Y_i = \bar{X} + \epsilon_i$$

where $$\epsilon_i \sim MVN(0,\Sigma)$$ is multivariate distributed and the covariance matrix is based on the residual maker matrix.

With this, any inference method on $$X_i$$ should work just as well with $$Y_i$$. So any inference method can be made to depend on the sufficient statistic only.

Computational: with the code below we simulate the situation above and compute the sample $$X_i$$ as a normal distributed sample and $$Y_i$$, the surrogate data, based on only the mean of the sample. Then we perform some inference that depends on more than the sufficient statistic by computing the median and see how the methods differ based on a histogram. The result is that both methods result in the same distribution. It shows how the inference with a method that is not based on the sufficient statistic can be turned into a method that is based on the sufficient statistic with equivalent performance. The performance is equivalent no matter what measure of performance you use because the sample distributions of the estimates are exactly the same. set.seed(1)
mu = 5
n = 10

sim = function() {
### true sample
x = rnorm(n,mu,1)

### sufficient statistic
xbar = mean(x)

### computations for epsilon
### independent from original sample
vmu = rep(0,n)
vx = rep(1,n)
sigma = diag(rep(1,n)) - vx %*% solve(t(vx) %*% vx) %*% t(vx)
epsilon = MASS::mvrnorm(1,vmu,sigma)

### surrogate data
### only depending on true sample
### via sufficient statistic
y = xbar + epsilon
return(c(median(x),median(y)))
}

sims = replicate(10^5, sim())

### plot histogram
hist(sims[1,], main = "regression based on original data", xlab = expression(hat(mu)), breaks = seq(2.5,7.5,0.1))
hist(sims[2,], main = "regression based on surrogate data \n with only sufficient statistic as input", xlab = expression(hat(mu)), breaks = seq(2.5,7.5,0.1))


### Does this make alternatives useless? No!

The above is not to say that methods that do not depend on the sufficient statistic are useless. Those other methods can be more robust to situations where the assumptions about the distribution are wrong (and the assumptions about the distribution being wrong has implications such as what the sufficient statistic is).

A lot of inference is also not performed with an explicit distribution in mind, sometimes people want to estimate a property of a distribution (like the mean or median) without an explicit definition of that distribution (and without a sufficient statistic).

The sufficient statistic also doesn't contain all the information. It is the sufficient statistic plus the assumed true distribution that describes the population. When this assumption is not present or wrong then the sufficient statistic is also not containing all the information that the sample contains about the parameters or statistics that describe a population distribution.