Why is sufficient statistics/data reduction normally taught in Statistics? In most upper-level classes on statistical inference, data reduction and sufficient statistics are normally taught, but without too much motivation.
I understand sufficient statistics are important for many theorems, such as the Lehmann–Scheffé theorem. However, I have also been told that there is the emphasis on data reduction because statisticians in the past did not have access to large computing resources and so that was why. What is the reasoning?
 A: This answer is an oversimplification, bound to criticism, but I also believe it carries the essence behind the reason sufficient statistics are useful: the motivation for a sufficient statistics is the possibility it gives us of assessing information on the entire population without the need of all the data.
Say you get your grade on an exam and you want to know how well you did compared to your classmates. If you are given a sample mean and variance, you can do this without asking everyone's grades. Isn't it cool?
A: What you've been told is certainly NOT true. Data reduction is as important as ever. See for example Donoho's work on Compressed Sensing and thresholding estimators. Wavelet estimators and regularised estimators also work similarly - aim is to compress data on as few coefficient as possible. There is a parallel too with the concept of simplicity - compression allows us to describe data with a model as simple as possible (but not too simple) - Minimum Description/Message Length theories follow those lines.
The concept of sufficiency is nearly as old as the concept of (modern) statistics. It has been defined by Ronald A. Fisher on his seminal 1922 paper. 
As you may read, a sufficient statistic is the one that summarises the whole of the relevant information provided by the sample. Mathematically, if $\theta$ is to be estimated, $T_1(X)$ a statistic which contains the whole of the information as to the value of $\theta$, and $T_2(X)$ any other statistic, then the surface of distribution of pairs $T_1(X)$ $T_2(X)$, for a given value of $\theta$, is such that for a given value of $T_1(X)$, the distribution of $T_2(X)$ does not involve $\theta$. When $T_1(X)$ is known, knowledge of the value of  $T_2(X)$ throws no further light upon the value of $\theta$. 
That is, once you know the value sufficient statistic for the parameter of the population to be estimated, you need no further information - you don't need to store/process any subset of your data. Everything that can be said about the data is compressed on this statistic. 
A: You are correct in suggesting that the availability of almost limitless computational resources means that the importance of data reduction is lessened. For example, resampling statistics, at one time too computationally expensive for practical use, allow the entire sample to be utilised directly without assumption of populations. However, data reduction and sufficient statistics remains just about as important as ever.
Data reduction allows you to see what the data say about the topic of interest without the overwhelming distraction of the data. (Something about forest and trees should go here, I suspect.) Choose a sufficient statistic relevant to the topic, and take extra care is there doesn't happen to be one.
