Overview of predictive modelling, machine learning, etc.

Question

My text Intuitive Biostatistics is a nonmathematical explanation of conventional statistics. Chapter 3 explains the basic mindset of statistics as analyzing a sample to make inferences from a population, or to fit a scientifically sensible model to find parameters to understand and compare.

For the next (fourth) edition, I want to add a short section explaining that there are other mindsets used in data analysis. I'd welcome a critique. Thanks!

This chapter has discussed the basic idea of statistics as used in many scientific and clicnical situations. You can think of this approach as using data from a sample to make inferences about a population. Alternatively, you can think about fitting and comparing understandable models in order to obtain parameter values that can be interpreted and compared.

Sometimes statistics is used with an entirely different goal: to predict future events, or to find patterns in data. In these situations, it is not always necessary to think about samples and populations, or to think about a model that expresses a scientific idea. Instead the goal is to simply find an equation or algorithm that makes reasonably correct predictons. Enter one set of data to obtain a rule for making a prediction, and evalutate these predictions with another set of data. This approach goes by many names including machine learning, neural networks, data mining and predictive modelling. Those terms don’t all mean exactly the same thing, but all describe approaches to data analysis that use an approach not covered in this chapter or anywhere in this book.

Answer 1

I don't understand this phrase:

In these situations, it is not always necessary to think about samples and populations, or to think about a model that expresses a scientific idea.

It doesn't make sense to me, because if I were to build a regression model I would still need to think about my samples and population. I don't understand why I should just plug my sample data into R and hope for the best without any idea my sample is about. The sentence doesn't add anything, it's confusing and technically incorrect.

Instead the goal is to simply find an equation or algorithm that makes reasonably correct predictons sounds doggy to me. What do you mean by reasonably correct? Your users are probably not very mathematical (otherwise they wouldn't buy your book), they would't understand anything like R2. To them, a model is either good or bad. I think you should rephrase it.

neutral networks. I think you should drop it, it doesn't add anything.

Maybe add some diagrams to illustrate your idea? A simple linear regression for plotting the expected gene expression against measured gene expression? A decision tree for classifying type of cancer in the clinical setting is also not bad.

Answer 2

Are not all (or nearly) all approaches in some sense trying to deduce something that generalizes and thus predicts what will happen? There is not so much of a distinction in this respect and it is not an entirely different goal. The sentence

In these situations, it is not always necessary to think about samples and populations, or to think about a model that expresses a scientific idea.

seems wrong. These things are still very important. However, I would agree that the emphasis in these areas is more on prediction rather than e.g. hypothesis testing (which you might say was about proving/disproving scientific ideas). The bit about finding an algorithm (equation less often) that makes reasonably good predictions is a key bit. Predictions are not always evaluated on different data (see cross-validation).

It may also be worthwhile to mention that some different terminology (e.g. "learning" instead of "fitting") has developed in these areas due to their historical origin, but that many of the same ideas and issues apply.

In fact, not commenting on some of the issues in prediction would seem like a major omission. E.g. in our first Phase 2 trial in 20 patients our drug has a huge efficacy, can we expect the same efficacy in Phase 3? Or we test 10 doses in a dose finding study and simply pick the best one in terms of the point estimate, should we expect to see the same efficacy in Phase 3? The overall trial failed to show that the drug works, but we looked at 20 subgroups and decided that in one of them the drug works. How likely is it that a new trial would show this? These questions involve many of the same issues as one gets in machine learning - the more naïve things I describe above (which are sort of cases of over-fitting on your training data) are avoided to some extent by the more reliable machine learning approaches.

Answer 3

A few thoughts beyond those from @Björn and @StudentT that wouldn't fit into a comment on either of their answers.

It seems that the distinction you are trying to draw is between testing hypotheses on data (traditional statistical inference, the topic of your book) and gleaning relationships from data (machine learning, not covered in your book). But that distinction can be hard to make.

For example, your book does seem to cover genome-wide association studies (GWAS). I would generally think of GWAS more in the latter category, gleaning relationships from data rather than testing pre-specified hypotheses. The issues of multiple comparisons that you cover so thoroughly in your book are essentially the same for GWAS as for many data-mining/machine-learning situations.

There's also a slight danger that the reader will interpret this as a distinction between traditional inference as causal inference versus machine learning as pattern recognition, even though traditional inference often has no more true information on causal relations than does machine learning. You certainly discuss the fallacy of such interpretation of traditional inference in the book, but it might be safer not even to raise this possibility in the reader's mind at this point.

Furthermore, there's the interesting take on hypothesis testing and prediction in Frank Harrell's Regression Modeling Strategies. In the introduction to this book with its emphasis on prediction, he argues (page 1, second edition):

Prediction could be considered a superset of hypothesis testing and estimation.

As an example (page 2):

It is often useful to think of effect estimates as differences between two predicted values from a model.

Or more generally (page 3):

Thus when one develops a reasonable multivariable predictive model, hypothesis testing and estimation of effects are byproducts of the fitted model. So predictive modeling is often desirable even when prediction is not the main goal.

So for your purpose it might be better to downplay this distinction as all-or-none, as hypothesis testing and prediction do not necessarily "have entirely different goal[s]." What you cover in your book will also be of great value to anyone trying to make sense of many machine learning approaches. You can point out instead how readers of your book will be able to apply what they have learned from you when they set out to explore the other end of the spectrum of data analysis methods.