1
$\begingroup$

This is a definition of the sufficient statistic from Wikipedia.

A statistic $t = T(X)$ is sufficient for underlying parameter $θ$ precisely if the conditional probability distribution of the data $X$, given the statistic $t = T(X)$, does not depend on the parameter $θ$, i.e., $$ \text{Pr}(x|t,\theta) = \text{Pr}(x|t) $$

I have a trouble to interpret the definition. I don't understand the meaning of the probability conditional on a "parameter" $\theta$. How can knowing the value of a parameter make a difference on the probability distribution? If $\theta$ is a "parameter", which is not a random variable so has no uncertainty, should $Pr(x|\theta) = Pr(x)$ always? For example, let's consider throwing an unfair dice. The probability of each face may be considered as a parameter. The result of the experiment throwing the dice does not depend on my knowledge on the probabilities of each face. The experiment is physically determined and my knowledge on the dice has nothing to do with the outcomes. So conditioning on $\theta$ is meaningless and we have trivially $\text{Pr}(x|t,\theta) = \text{Pr}(x|t)$ for any $t$ and $\theta$. What is my misconception here?

$\endgroup$
1
  • $\begingroup$ In essence, your description below the quote is that of a sufficient statistic, so perhaps your question is what is not a sufficient statistic? $\endgroup$ – Frans Rodenburg Oct 24 '18 at 4:36
2
$\begingroup$

If your issue is with the use of a conditioning symbol "|" instead of a semi-colon or an index in the notation of a family of distributions $$\text{Pr}(x|\theta)\qquad\text{vs.}\qquad \text{Pr}(x;\theta) \qquad \text{vs.}\qquad\text{Pr}_\theta(x)$$ a first answer is that it is a matter of notations and that once one has clearly set a notation for this family it can be used as such. (Notation-wise, using $\text{Pr(x)}$ for a distribution should be discouraged as it does not apply to continuous and mixed random variables.) The distribution (and the associated density) are dependent on the value of $\theta$, meaning that two different $\theta$'s [should] lead to two different densities. This function is thus defined conditional on the chosen value of $\theta$ and one could not use it without knowing this value of $\theta$.

A second answer is that a conditional distribution sets the conditioning variable to a fixed value. In a conditional density $f(x|y)$ the value $y$ is constant while $x$ varies on the state space $\mathcal{X}$, e.g., $\mathbb{R}$. The value of $y$ taken by the conditioning variable $Y$ thus determines the function [of $x$] and for all purposes acts like a parameter. There is thus no [pragmatic] distinction to be made between $y$ and a parameter $\theta$ when using this conditional density. Furthermore, if $\theta$ becomes random, as for instance in a Bayesian analysis, adopting the conditional sign on $\theta$ makes notations coherent [with a joint distribution on $x$ and $\theta$].

$\endgroup$
1
$\begingroup$

First of all the distribution of x is not actually a distribution but a family of distributions. You know assume a form of x which is a function of $\theta$ but you do not know the value of $\theta$.

This is similar with the family of normal distributions $N(\mu,\sigma^2)$ where you don’t know the value of parameters versus standard normal distribution $N(0,1)$. Now what means $P(x|\theta)$ ? This is actually the distribution function, remember that the conditional formula is read “probability of x given the value of theta”.

Now the formula for sufficient statistic says if you know the value of statistic and the one of the parameter the distribution you determine for x is the same as the one determined for x if you knew the value of statistic only. So knowing the statistic is enough (sufficient) to determine the distribution of x. Or otherwise said, the statistic can be used to estimate the parameter entirely, without a need for additional information.

In your example with dice you assume a distribution family multinoulli, but you do not know the parameters. If you estimate the parameters values with a statistic and you determine the same distribution for the dice as if you would knew the parameters value then the statistic used for inference is sufficient.

One final word about what means determine in the previous paragraph. Of course you cannot determine exactly the parameters values, the statistic is built from sample, so it contains limited information. The determination here refers to the procedure itself. Everything is valid at limit, if you would have an infinite sample and so on.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.