The Two Cultures: statistics vs. machine learning?

Question

Last year, I read a blog post from Brendan O'Connor entitled "Statistics vs. Machine Learning, fight!" that discussed some of the differences between the two fields. Andrew Gelman responded favorably to this:

Simon Blomberg:

From R's fortunes package: To paraphrase provocatively, 'machine learning is statistics minus any checking of models and assumptions'. -- Brian D. Ripley (about the difference between machine learning and statistics) useR! 2004, Vienna (May 2004) :-) Season's Greetings!

Andrew Gelman:

In that case, maybe we should get rid of checking of models and assumptions more often. Then maybe we'd be able to solve some of the problems that the machine learning people can solve but we can't!

There was also the "Statistical Modeling: The Two Cultures" paper by Leo Breiman in 2001 which argued that statisticians rely too heavily on data modeling, and that machine learning techniques are making progress by instead relying on the predictive accuracy of models.

Has the statistics field changed over the last decade in response to these critiques? Do the two cultures still exist or has statistics grown to embrace machine learning techniques such as neural networks and support vector machines?

Answer 1

I think the answer to your first question is simply in the affirmative. Take any issue of Statistical Science, JASA, Annals of Statistics of the past 10 years and you'll find papers on boosting, SVM, and neural networks, although this area is less active now. Statisticians have appropriated the work of Valiant and Vapnik, but on the other side, computer scientists have absorbed the work of Donoho and Talagrand. I don't think there is much difference in scope and methods any more. I have never bought Breiman's argument that CS people were only interested in minimizing loss using whatever works. That view was heavily influenced by his participation in Neural Networks conferences and his consulting work; but PAC, SVMs, Boosting have all solid foundations. And today, unlike 2001, Statistics is more concerned with finite-sample properties, algorithms and massive datasets.

But I think that there are still three important differences that are not going away soon.

1. Methodological Statistics papers are still overwhelmingly formal and deductive, whereas Machine Learning researchers are more tolerant of new approaches even if they don't come with a proof attached;
2. The ML community primarily shares new results and publications in conferences and related proceedings, whereas statisticians use journal papers. This slows down progress in Statistics and identification of star researchers. John Langford has a nice post on the subject from a while back;
3. Statistics still covers areas that are (for now) of little concern to ML, such as survey design, sampling, industrial Statistics etc.

Answer 2

The biggest difference I see between the communities is that statistics emphasizes inference, whereas machine learning emphasized prediction. When you do statistics, you want to infer the process by which data you have was generated. When you do machine learning, you want to know how you can predict what future data will look like w.r.t. some variable.

Of course the two overlap. Knowing how the data was generated will give you some hints about what a good predictor would be, for example. However, one example of the difference is that machine learning has dealt with the p >> n problem (more features/variables than training samples) since its infancy, whereas statistics is just starting to get serious about this problem. Why? Because you can still make good predictions when p >> n, but you can't make very good inferences about what variables are actually important and why.

Answer 3

Bayesian: "Hello, Machine Learner!"

Frequentist: "Hello, Machine Learner!"

Machine Learner: "I hear you guys are good at stuff. Here's some data."

F: "Yes, let's write down a model and then calculate the MLE."

B: "Hey, F, that's not what you told me yesterday! I had some univariate data and I wanted to estimate the variance, and I calculated the MLE. Then you pounced on me and told me to divide by $n-1$ instead of by $n$."

F: "Ah yes, thanks for reminding me. I often think that I'm supposed to use the MLE for everything, but I'm interested in unbiased estimators and so on."

ML: "Eh, what's this philosophizing about? Will it help me?"

F: " OK, an estimator is a black box, you put data in and it gives you some numbers out. We frequentists don't care about how the box was constructed, about what principles were used to design it. For example, I don't know how to derive the $\div(n-1)$ rule."

ML: " So, what do you care about?"

F: "Evaluation."

ML: "I like the sound of that."

F: "A black box is a black box. If somebody claims a particular estimator is an unbiased estimator for $\theta$, then we try many values of $\theta$ in turn, generate many samples from each based on some assumed model, push them through the estimator, and find the average estimated $\theta$. If we can prove that the expected estimate equals the true value, for all values, then we say it's unbiased."

ML: "Sounds great! It sounds like frequentists are pragmatic people. You judge each black box by its results. Evaluation is key."

F: "Indeed! I understand you guys take a similar approach. Cross-validation, or something? But that sounds messy to me."

ML: "Messy?"

F: "The idea of testing your estimator on real data seems dangerous to me. The empirical data you use might have all sorts of problems with it, and might not behave according the model we agreed upon for evaluation."

ML: "What? I thought you said you'd proved some results? That your estimator would always be unbiased, for all $\theta$."

F: "Yes. While your method might have worked on one dataset (the dataset with train and test data) that you used in your evaluation, I can prove that mine will always work."

ML: "For all datasets?"

F: "No."

ML: "So my method has been cross-validated on one dataset. You haven't test yours on any real dataset?"

F: "That's right."

ML: "That puts me in the lead then! My method is better than yours. It predicts cancer 90% of the time. Your 'proof' is only valid if the entire dataset behaves according to the model you assumed."

F: "Emm, yeah, I suppose."

ML: "And that interval has 95% coverage. But I shouldn't be surprised if it only contains the correct value of $\theta$ 20% of the time?"

F: "That's right. Unless the data is truly i.i.d Normal (or whatever), my proof is useless."

ML: "So my evaluation is more trustworthy and comprehensive? It only works on the datasets I've tried so far, but at least they're real datasets, warts and all. There you were, trying to claim you were more 'conservative' and 'thorough' and that you were interested in model-checking and stuff."

B: (interjects) "Hey guys, Sorry to interrupt. I'd love to step in and balance things up, perhaps demonstrating some other issues, but I really love watching my frequentist colleague squirm."

F: "Woah!"

ML: "OK, children. It was all about evaluation. An estimator is a black box. Data goes in, data comes out. We approve, or disapprove, of an estimator based on how it performs under evaluation. We don't care about the 'recipe' or 'design principles' that are used."

F: "Yes. But we have very different ideas about which evaluations are important. ML will do train-and-test on real data. Whereas I will do an evaluation that is more general (because it involves a broadly-applicable proof) and also more limited (because I don't know if your dataset is actually drawn from the modelling assumptions I use while designing my evaluation.)"

ML: "What evaluation do you use, B?"

F (interjects): "Hey. Don't make me laugh. He doesn't evaluate anything. He just uses his subjective beliefs and runs with it. Or something."

B: "That's the common interpretation. But it's also possible to define Bayesianism by the evaluations preferred. Then we can use the idea that none of us care what's in the black box, we care only about different ways to evaluate."

B continues: "Classic example: Medical test. The result of the blood test is either Positive or Negative. A frequentist will be interested in, of the Healthy people, what proportion get a Negative result. And similarly, what proportion of Sick people will get a Positive. The frequentist will calculate these for each blood testing method that's under consideration and then recommend that we use the test that got the best pair of scores."

F: "Exactly. What more could you want?"

B: "What about those individuals that got a Positive test result? They will want to know 'of those that get a Positive result, how many will get Sick?' and 'of those that get a Negative result, how many are Healthy?' "

ML: "Ah yes, that seems like a better pair of questions to ask."

F: "HERESY!"

B: "Here we go again. He doesn't like where this is going."

ML: "This is about 'priors', isn't it?"

F: "EVIL".

B: "Anyway, yes, you're right ML. In order to calculate the proportion of Positive-result people that are Sick you must do one of two things. One option is to run the tests on lots of people and just observe the relevant proportions. How many of those people go on to die of the disease, for example."

ML: "That sounds like what I do. Use train-and-test."

B: "But you can calculate these numbers in advance, if you are willing to make an assumption about the rate of Sickness in the population. The frequentist also makes his calculations in advance, but without using this population-level Sickness rate."

F: "MORE UNFOUNDED ASSUMPTIONS."

B: "Oh shut up. Earlier, you were found out. ML discovered that you are just as fond of unfounded assumptions as anyone. Your 'proven' coverage probabilities won't stack up in the real world unless all your assumptions stand up. Why is my prior assumption so different? You call me crazy, yet you pretend your assumptions are the work of a conservative, solid, assumption-free analysis."

B (continues): "Anyway, ML, as I was saying. Bayesians like a different kind of evaluation. We are more interested in conditioning on the observed data, and calculating the accuracy of our estimator accordingly. We cannot perform this evaluation without using a prior. But the interesting thing is that, once we decide on this form of evaluation, and once we choose our prior, we have an automatic 'recipe' to create an appropriate estimator. The frequentist has no such recipe. If he wants an unbiased estimator for a complex model, he doesn't have any automated way to build a suitable estimator."

ML: "And you do? You can automatically build an estimator?"

B: "Yes. I don't have an automatic way to create an unbiased estimator, because I think bias is a bad way to evaluate an estimator. But given the conditional-on-data estimation that I like, and the prior, I can connect the prior and the likelihood to give me the estimator."

ML: "So anyway, let's recap. We all have different ways to evaluate our methods, and we'll probably never agree on which methods are best."

B: "Well, that's not fair. We could mix and match them. If any of us have good labelled training data, we should probably test against it. And generally we all should test as many assumptions as we can. And some 'frequentist' proofs might be fun too, predicting the performance under some presumed model of data generation."

F: "Yeah guys. Let's be pragmatic about evaluation. And actually, I'll stop obsessing over infinite-sample properties. I've been asking the scientists to give me an infinite sample, but they still haven't done so. It's time for me to focus again on finite samples."

ML: "So, we just have one last question. We've argued a lot about how to evaluate our methods, but how do we create our methods."

B: "Ah. As I was getting at earlier, we Bayesians have the more powerful general method. It might be complicated, but we can always write some sort of algorithm (maybe a naive form of MCMC) that will sample from our posterior."

F (interjects): "But it might have bias."

B: "So might your methods. Need I remind you that the MLE is often biased? Sometimes, you have great difficulty finding unbiased estimators, and even when you do you have a stupid estimator (for some really complex model) that will say the variance is negative. And you call that unbiased. Unbiased, yes. But useful, no!"

ML: "OK guys. You're ranting again. Let me ask you a question, F. Have you ever compared the bias of your method with the bias of B's method, when you've both worked on the same problem?"

F: "Yes. In fact, I hate to admit it, but B's approach sometimes has lower bias and MSE than my estimator!"

ML: "The lesson here is that, while we disagree a little on evaluation, none of us has a monopoly on how to create estimator that have properties we want."

B: "Yes, we should read each other's work a bit more. We can give each other inspiration for estimators. We might find that other's estimators work great, out-of-the-box, on our own problems."

F: "And I should stop obsessing about bias. An unbiased estimator might have ridiculous variance. I suppose all of us have to 'take responsibility' for the choices we make in how we evaluate and the properties we wish to see in our estimators. We can't hide behind a philosophy. Try all the evaluations you can. And I will keep sneaking a look at the Bayesian literature to get new ideas for estimators!"

B:"In fact, a lot of people don't really know what their own philosophy is. I'm not even sure myself. If I use a Bayesian recipe, and then proof some nice theoretical result, doesn't that mean I'm a frequentist? A frequentist cares about above proofs about performance, he doesn't care about recipes. And if I do some train-and-test instead (or as well), does that mean I'm a machine-learner?"

ML: "It seems we're all pretty similar then."

Answer 4

In such a discussion, I always recall the famous Ken Thompson quote

When in doubt, use brute force.

In this case, machine learning is a salvation when the assumptions are hard to catch; or at least it is much better than guessing them wrong.

Answer 5

What enforces more separation than there should be is each discipline's lexicon.

There are many instances where ML uses one term and Statistics uses a different term--but both refer to the same thing--fine, you would expect that, and it doesn't cause any permanent confusion (e.g., features/attributes versus expectation variables, or neural network/MLP versus projection-pursuit).

What's much more troublesome is that both disciplines use the same term to refer to completely different concepts.

A few examples:

Kernel Function

In ML, kernel functions are used in classifiers (e.g., SVM) and of course in kernel machines. The term refers to a simple function (cosine, sigmoidal, rbf, polynomial) to map non-linearly separable to a new input space, so that the data is now linearly separable in this new input space. (versus using a non-linear model to begin with).

In statistics, a kernel function is weighting function used in density estimation to smooth the density curve.

Regression

In ML, predictive algorithms, or implementations of those algorithms that return class labels "classifiers" are (sometimes) referred to as machines--e.g., support vector machine, kernel machine. The counterpart to machines are regressors, which return a score (continuous variable)--e.g., support vector regression.

Rarely do the algorithms have different names based on mode--e.g., a MLP is the term used whether it returns a class label or a continuous variable.

In Statistics, regression, if you are attempting to build a model based on empirical data, to predict some response variable based on one or more explanatory variables or more variables--then you are doing regression analysis. It doesn't matter whether the output is a continuous variable or a class label (e.g., logistic regression). So for instance, least-squares regression refers to a model that returns a continuous value; logistic regression on the other hand, returns a probability estimate which is then discretized to a class labels.

Bias

In ML, the bias term in the algorithm is conceptually identical to the intercept term used by statisticians in regression modeling.

In Statistics, bias is non-random error--i.e., some phenomenon influenced the entire data set in the same direction, which in turn means that this kind of error cannot be removed by resampling or increasing the sample size.

Answer 6

The largest differences I've been noticing in the past year are:

• Machine learning experts do not spend enough time on fundamentals, and many of them do not understand optimal decision making and proper accuracy scoring rules. They do not understand that predictive methods that make no assumptions require larger sample sizes than those that do.
• We statisticians spend too little time learning good programming practice and new computational languages. We are too slow to change when it comes to computing and adopting new methods from the statistical literature.

Answer 7

Machine Learning seems to have its basis in the pragmatic - a Practical observation or simulation of reality. Even within statistics, mindless "checking of models and assumptions" can lead to discarding methods that are useful.

For example, years ago, the very first commercially available (and working) Bankruptcy model implemented by the credit bureaus was created through a plain old linear regression model targeting a 0-1 outcome. Technically, that's a bad approach, but practically, it worked.

Answer 8

I disagree with this question as it suggests that machine learning and statistics are different or conflicting sciences.... when the opposite is true!

machine learning makes extensive use of statistics... a quick survey of any Machine learning or data mining software package will reveal Clustering techniques such as k-means also found in statistics.... will also show dimension reduction techniques such as Principal components analysis also a statistical technique... even logistic regression yet another.

In my view the main difference is that traditionally statistics was used to proove a pre conceived theory and usually the analysis was design around that principal theory. Where with data mining or machine learning the opposite approach is usually the norm in that we have the outcome we just want to find a way to predict it rather than ask the question or form the theory is this the outcome!

Answer 9

The real problem is that this question is misguided. It is not machine learning vs statistics, it is machine learning against real scientific advance. If a machine learning device gives the right predictions 90% of the time but I cannot understand "why", what is the contribution of machine learning to science at large? Imagine if machine learning techniques were used to predict the positions of planets: there would be a lot of smug people thinking that they can accurately predict a number of things with their SVMs, but what would they really know about the problem they have in their hands? Obviously, science does not really advance by numerical predictions, it advances by means of models (mental, mathematical) who let us see far beyond than just numbers.

Answer 10

I have spoken on this at a different forum the ASA Statistical Consulting eGroup. My response was more specifically to data mining but the two go hand in hand. We statisticians have snubbed our noses at data miners, computer scientists, and engineers. It is wrong. I think part of the reason it happens is because we see some people in those fields ignoring the stochastic nature of their problem. Some statisticians call data mining data snooping or data fishing. Some people do abuse and misuse the methods but statisticians have fallen behind in data mining and machine learning because we paint them with a broad brush. Some of the big statistical results have come from outside the field of statistics. Boosting is one important example. But statisticians like Breiman, Friedman, Hastie, Tibshirani, Efron, Gelman and others got it and their leadership has brought statisticians into the analysis of microarrays and other large scale inference problems. So while the cultures may never mesh there is now more cooperation and collaboration between the computer scientists, engineers and statisticians.

Answer 11

Statistical learning (AKA Machine Learning) has its origins in the quest to create software by "learning from examples". There are many tasks that we would like computers to do (e.g., computer vision, speech recognition, robot control) that are difficult to program but for which it is easy to provide training examples. The machine learning/statistical learning research community developed algorithms to learn functions from these examples. The loss function was typically related to the performance task (vision, speech recognition). And of course we had no reason to believe there was any simple "model" underlying these tasks (because otherwise we would have coded up that simple program ourselves). Hence, the whole idea of doing statistical inference didn't make any sense. The goal is predictive accuracy and nothing else.

Over time, various forces started driving machine learning people to learn more about statistics. One was the need to incorporate background knowledge and other constraints on the learning process. This led people to consider generative probabilistic models, because these make it easy to incorporate prior knowledge through the structure of the model and priors on model parameters and structure. This led the field to discover the rich statistical literature in this area. Another force was the discovery of the phenomenon of overfitting. This led the ML community to learn about cross-validation and regularization and again we discovered the rich statistical literature on the subject.

Nonetheless, the focus of most machine learning work is to create a system that exhibits certain performance rather than the make inferences about an unknown process. This is the fundamental difference between ML and statistics.

Answer 12

Ideally one should have a thorough knowledge of both statsitics and machine learning before attempting to answer his question. I am very much a neophyte to ML, so forgive me if wat I say is naive.

I have limited experience in SVMs and regression trees. What strikes me as lacking in ML from a stats point of view is a well developed concept of inference.

Inference in ML seems to boil down almost exclusively to the predictice accuracy, as measured by (for example) mean classification error (MCE), or balanced error rate (BER) or similar. ML is in the very good habit of dividing data randomly (usually 2:1) into a training set and a test set. Models are fit using the training set and performance (MCE, BER etc) is assessed using the test set. This is an excellent practice and is only slowly making its way into mainstream statistics.

ML also makes heavy use of resampling methods (especially cross-validation), whose origins appear to be in statistics.

However, ML seems to lack a fully developed concept of inference - beyond predictive accuracy. This has two results.

1) There does not seem to be an appreciation that any prediction (parameter estimation etc.) is subject to a random error and perhaps systemmatics error (bias). Statisticians will accept that this is an inevitable part of prediction and will try and estimate the error. Statistical techniques will try and find an estimate that has minimum bias and random error. Their techniques are usually driven by a model of the data process, but not always (eg. Bootstrap).

2) There does not seem to be a deep understanding in ML of the limits of applying a model to new data to a new sample from the same population (in spite of what I said earlier about the training-test data set approach). Various statistical techniques, among them cross validation and penalty terms applied to likelihood-based methods, guide statisticians in the trade-off between parsimony and model complexity. Such guidelines in ML seem much more ad hoc.

I've seen several papers in ML where cross validation is used to optimise a fitting of many models on a training dataset - producing better and better fit as the model complexity increases. There appears little appreciation that the tiny gains in accuracy are not worth the extra complexity and this naturally leads to over-fitting. Then all these optimised models are applied to the test set as a check on predictive performance and to prevent overfitting. Two things have been forgotten (above). The predictive performance will have a stochastic component. Secondly multiple tests against a test set will again result in over-fitting. The "best" model will be choisen by the ML practitioner without a full appreciation he/she has cherry picked from one realisation of many possible outomes of this experiment. The best of several tested models will almost certainly not reflect the true performance on new data.

Any my 2 cents worth. We have much to learn from each other.

Answer 13

This question can also be extended to the so-called super-culture of data science in 2015 David Donoho paper 50 years of Data Science, where he confronts different points of view from statistics and computer science (including machine learning), for instance direct standpoints (from different persons) such that:

• Why Do We Need Data Science When We've Had Statistics for Centuries?
• Data Science is statistics.
• Data Science without statistics is possible, even desirable.
• Statistics is the least important part of data science.

and assorted with historical, philosophical considerations, for instance:

It is striking how, when I review a presentation on today's data science, in which statistics is supercially given pretty short shrift, I can't avoid noticing that the underlying tools, examples, and ideas which are being taught as data science were all literally invented by someone trained in Ph.D. statistics, and in many cases the actual software being used was developed by someone with an MA or Ph.D. in statistics. The accumulated eorts of statisticians over centuries are just too overwhelming to be papered over completely, and can't be hidden in the teaching, research, and exercise of Data Science.

This essay has generated many responses and contributions to the debate.

Answer 14

I don't really know what the conceptual/historical difference between machine learning and statistic is but I am sure it is not that obvious... and I am not really interest in knowing if I am a machine learner or a statistician, I think 10 years after Breiman's paper, lots of people are both...

Anyway, I found interesting the question about predictive accuracy of models. We have to remember that it is not always possible to measure the accuracy of a model and more precisely we are most often implicitly making some modeling when measuring errors.

For Example, mean absolute error in time series forecast is a mean over time and it measures the performance of a procedure to forecast the median with the assumption that performance is, in some sense, stationary and shows some ergodic property. If (for some reason) you need to forecast the mean temperature on earth for the next 50 years and if your modeling performs well for the last 50 years... it does not means that...

More generally, (if I remember, it is called no free lunch) you can't do anything without modeling... In addition, I think statistic is trying to find an answer to the question : "is something significant or not ", this is a very important question in science and can't be answered through a learning process. To state John Tukey (was he a statistician ?) :

The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data

Hope this helps !

Answer 15

Clearly, the two fields clearly face similar but different problems, in similar but not identical ways with analogous but not identical concepts, and work in different departments, journals and conferences.

When I read Cressie and Read's Power Divergence Statistic it all snapped into place for me. Their formula generalizes commonly used test statistics into one that varies by one exponent, lambda. There are two special cases, lambda=0 and lambda=1.

Computer Science and Statistics fit along a continuum (that presumably could include other points). At one value of lambda, you get statistics commonly cited in Statistics circles, and at the other you get statistics commonly cited in Comp Sci circles.

Statistics

• Lambda = 1
• Sums of squares appear a lot
• Variance as a measure of variability
• Covariance as a measure of association
• Chi-squared statistic as a measure of model fit

Computer science:

• Lambda = 0
• Sums of logs appear a lot
• Entropy as a measure of variability
• Mutual information as a measure of association
• G-squared statistic as a measure of model fit

Answer 16

You run a fancy computer algorithm once -- and you get a CS conference presentation/statistics paper (wow, what a fast convergence!). You commercialize it and run it 1 million times -- and you go broke (ouch, why am I getting useless and irreproducible results all the time???) unless you know how to employ probability and statistics to generalize the properties of the algorithm.

Answer 17

There is an area of application of statistics where focus on the data generating model makes a lot of sense. In designed experiments, e.g., animal studies, clinical trials, industrial DOEs, statisticians can have a hand in what the data generating model is. ML tends not to spend much time on this very important problem as ML usually focuses on another very important problem of prediction based on “large” observational data. That is not to say that ML can’t be applied to “large” designed experiments, but it is important to acknowledge that statistics has particular expertise on “small” data problems arising from resource constrained experiments.

At the end of the day I think we can all agree to use what works best to solve the problem at hand. E.g., we may have a designed experiment that produces very wide data with the goal of prediction. Statistical design principles are very useful here and ML methods could be useful to build the predictor.

Answer 18

I think machine learning needs to be a sub-branch under statistics, just like, in my view, chemistry needs to be a sub-branch under physics.

I think physics-inspired view into chemistry is pretty solid (I guess). I don't think there is any chemical reaction whose equivalent is not known in physical terms. I think physics has done an amazing job by explaining everything we can see at a chemistry level. Now the physicists' challenge seems to be explaining the tiny mysteries at the quantum level, under extreme conditions that are not observable.

Now back to machine learning. I think it too should be a sub-branch under statistics (just how chemistry is a sub-branch of physics).

But it seems to me that, somehow, either the current state of machine learning, or statistics, is not mature enough to perfectly realize this. But in the long run, I think one must become a sub-branch of the other. I think it's ML that will to get under statistics.

I personally think that "learning" and "analyzing samples" to estimate/infer functions or predictions are all essentially a question of statistics.

Answer 19

From the Coursera course "Data Science in real life" by Brian Caffo

Machine Learning

• Emphasize predictions
• Evaluates results via prediction performance
• Concern for overfitting but not model complexity per se
• Emphasis on performance
• Generalizability is obtained through performance on novel datasets
• Usually, no superpopulation model specified
• Concern over performance and robustness

Traditional statistical analysis

• Emphasizes superpopulation inference
• Focuses on a-priori hypotheses
• Simpler models preferred over complex ones (parsimony), even if the more complex models perform slightly better
• Emphasis on parameter interpretability
• Statistical modeling or sampling assumptions connects data to a population of interest
• Concern over assumptions and robustness

Answer 20

As as Computer Scientist, I am always intrigued when looking to statistical approaches. To me many times it looks like the statistical models used in the statistical analysis are way too complex for the data in many situations!

For example there is a strong link between data compression and statistics. Basically one needs a good statistical model which is able to predict the data well and this brings a very good compression of the data. In computer science when compressing the data always the complexity of the statistical model and the accuracy of the prediction are very important. Nobody wants to get have EVER a data file (containing sound data or image data or video data) becoming bigger after the compression!

I find that there are more dynamic things in computer science regarding statistics, like for example Minimum Description Length and Normalized Maximum Likelihood.