I have been reading about sufficient statistics, and I have a few questions whose answer I cannot find in the book.
Why do we care about sufficient statistics in the first place? My understanding is that we would like to use sufficient statistics to do inference on the parameter $\theta$. However, can't we just use the original sample? Or is it because long time ago we did not have powerful computers?
Strangely, I cannot find any discussion on the existence of sufficient statistics/minimal sufficient statistics. Is it trivial that given a random sample $X_1,\cdots,X_n \sim f(x\mid \theta)$, we can always find a SS that is not the whole sample? Is it true that we can always find a MSS?
How do we actually use SS/MSS to infer $\theta$? I read about the moment method as well as the maximum-likelihood method, but they just use the whole sample instead of a sufficient statistics $T(X_1,\cdots,X_n)$.
Most of the textbooks formulate the idea of sufficient statistics using conditional probability. For example, something like this,
Suppose $X_1,\cdots,X_n \sim f(x\mid \theta). $A statistics is sufficient if $$\mathbb{P}(X_1,\cdots,X_n \mid T(X_1,\cdots,X_n))$$ does not depend on $\theta$.
Purely out of curiosity, is it possible to define SS using, say, information theory or entropy? I am not an expert on this topic, but "$T$ contains all information from the sample that we need to determine $\theta$", or "$T$ contains as much information as the sample does to infer $\theta$" makes me think of information theory.