Several questions about sufficient statistics

Question

I have been reading about sufficient statistics, and I have a few questions whose answer I cannot find in the book.

1. Why do we care about sufficient statistics in the first place? My understanding is that we would like to use sufficient statistics to do inference on the parameter $\theta$. However, can't we just use the original sample? Or is it because long time ago we did not have powerful computers?

2. Strangely, I cannot find any discussion on the existence of sufficient statistics/minimal sufficient statistics. Is it trivial that given a random sample $X_1,\cdots,X_n \sim f(x\mid \theta)$, we can always find a SS that is not the whole sample? Is it true that we can always find a MSS?

3. How do we actually use SS/MSS to infer $\theta$? I read about the moment method as well as the maximum-likelihood method, but they just use the whole sample instead of a sufficient statistics $T(X_1,\cdots,X_n)$.

4. Most of the textbooks formulate the idea of sufficient statistics using conditional probability. For example, something like this,

Suppose $X_1,\cdots,X_n \sim f(x\mid \theta). $A statistics is sufficient if $$\mathbb{P}(X_1,\cdots,X_n \mid T(X_1,\cdots,X_n))$$ does not depend on $\theta$.

Purely out of curiosity, is it possible to define SS using, say, information theory or entropy? I am not an expert on this topic, but "$T$ contains all information from the sample that we need to determine $\theta$", or "$T$ contains as much information as the sample does to infer $\theta$" makes me think of information theory.

Answer 1

1. Efficiency is a big factor here, and will always be regardless of how fast computers get. Say you want to fit a normal distribution to some 1D data $\{ X_1, \ldots X_N\}$, but we know that in the future we may want to add more data to this dataset and update our fit. One way to do this is store all the data and recalculate whenever new data arrives - but that means our storage and time taken to calculate the parameters will grow $O(N)$. If instead we just store $\sum_{i=1}^N X_i$, $\sum_{i=1}^N X^2_i$ and $N$, then we have all the data we need to update our parameter estimates later when new data arrives, and our storage and runtime only grow $O(1)$. Depending on the application and dataset sizes involved that can be a make or break issue.
2. I'm not really familiar with this, but looking on Wikipedia it seems that "...according to the Pitman–Koopman–Darmois theorem, among families of probability distributions whose domain does not vary with the parameter being estimated, only in exponential families is there a sufficient statistic whose dimension remains bounded as sample size increases". My gut feeling has always been that the existance of sufficient statistics in a problem is usually a pretty special case and indicates a "nice" distribution. A lot of the time I'm working with whole datasets instead.
3. It depends on the problem. Generally, once you've found the factorisation $f_\theta(x) = h(x)g_\theta(T(x))$, to estimate $\theta$ you can just throw away the $h(x)$ part and just use the $g_\theta(T(x))$ to do whatever you like - so, if you're doing max likelihood, you just need to differentiate and set to zero $g_\theta(T(x))$ instead of $f_\theta(x)$, which usually ought to make the problem easier. I find that's really my main use of sufficient stats day to day - not as an answer to a problem in and of themselves, but as a way to simplify and speed up steps as part of another solution.