Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Suppose that we have a parameter of $k$-dimensions. Say, for example, for $N(u,\theta)$ both unknown then the parameter is of two dimensions, and $n$ i.i.d. observations.
Is it possible to find a sufficient statistic that has dimension less than $k$? For example, for normal distribution the minimal sufficient statistics is proven to be (sample mean, sample variance) so it is not possible for normal distribution, but I am wondering if this holds in general.
Since sufficiency of fixed dimension only occurs in exponential families (Darmois-Pitman-Koopman lemma), apart from distributions with varying support like the Uniform, let us consider an exponential family with parameter $\theta$ and density [against a fixed dominating measure] $$f_\theta(x)=\exp\left\{\sum_{i=1}^k a_i(\theta) T_i(x) -\psi(\theta)\right\}$$ Assuming the functions $a_i$ are linearly independent on the maximal support of $\theta$ (namely, the range of $\theta$'s for which the density is integrable), the model associated with this density can be reparameterised in $\alpha=(\alpha_1,\ldots,\alpha_k)$ which varies in (at least) the parameter space $$A=\left\{\alpha=(\alpha_1,\ldots,\alpha_k);\,\exists\theta\in\Theta, \alpha_i=a_i(\theta) \right\}$$at least, since the parameter space can be naturally expanded to its natural limit $$A=\left\{\alpha=(\alpha_1,\ldots,\alpha_k);\,\displaystyle{\int \exp\left\{\sum_{i=1}^k \alpha_i T_i(x)\right\}\text{d}\lambda(x)}<\infty \right\}\,,$$and that the functions $T_i$ are also linearly independent over the support $\cal X$ of $X$, the statistic $$T=(T_1(X),\ldots,T_k(X)$$is sufficient and with density $$g_\alpha(t)=\exp\left\{\sum_{i=1}^k \alpha_i t_i-\mu(\alpha)\right\}$$ against the appropriate dominating measure. Therefore on the natural space of the exponential family (which may be larger than the original parameter space), the sufficient statistic is of the same dimension as the parameter. Even though the domain of variation of $T(x)$ can be constrained by non-linear relations, there is a sample size after which the dimensional constraint vanishes.
A completely different approach, avoiding exponential families, is provided by
Edward W. Barankin and Melvin Katz, Jr.
Sufficient Statistics of Minimal Dimension
Sankhyā: The Indian Journal of Statistics Vol. 21, No. 3/4 (Aug., 1959), pp. 217-246
and they show the following result
where $r$ is the dimension of the sufficient statistic $T$ and $\rho(x^0)$ is the (local) rank of the second derivative of the log-likelihood in $\theta$ and $x$ [the definition is a bit too intricate to be reproduced here].
