What is wrong with extrapolation?

Question

I remember sitting in stats courses as an undergrad hearing about why extrapolation was a bad idea. Furthermore, there are a variety of sources online which comment on this. There's also a mention of it here.

Can anyone help me understand why extrapolation is a bad idea? If it is, how is it that forecasting techniques aren't statistically invalid?

Answer 1

A regression model is often used for extrapolation, i.e. predicting the response to an input which lies outside of the range of the values of the predictor variable used to fit the model. The danger associated with extrapolation is illustrated in the following figure.

The regression model is “by construction” an interpolation model, and should not be used for extrapolation, unless this is properly justified.

Answer 2

This xkcd comic explains it all.

Using the data points Cueball (the man with the stick) has, he has extrapolated that the woman will have "four dozen" husbands by late next month, and used this extrapolation to lead to the conclusion of buying the wedding cake in bulk.

Edit 3: For those of you who say "he doesn't have enough data points", here's another xkcd comic:

Here, the usage of the word "sustainable" over time is shown on a semi-log plot, and extrapolating the data points we receive an unreasonable estimates of how often the word "sustainable" will occur in the future.

Edit 2: For those of you who say "you need all past data points too", yet another xkcd comic:

Here, we have all past data points but we fail to accurately predict the resolution of Google Earth. Note that this is a semi-log graph too.

Edit: Sometimes, even the strongest of (r=.9979 in this case) correlations are just plain wrong.

---

If you extrapolate without other supporting evidence you also violating correlation does not imply causation; another great sin in the world of statistics.

If you do extrapolate X with Y, however, you must make sure that you can accurately (enough to satisfy your requirements) predict X with only Y. Almost always, there are multiple factors than impact X.

_{I would like to share a link to another answer that explains it in the words of Nassim Nicholas Taleb.}

Answer 3

"Prediction is very difficult, especially if it's about the future". The quote is attributed to many people in some form. I restrict in the following "extrapolation" to "prediction outside the known range", and in a one-dimensional setting, extrapolation from a known past to an unknown future.

So what is wrong with extrapolation. First, it is not easy to model the past. Second, it is hard to know whether a model from the past can be used for the future. Behind both assertions dwell deep questions about causality or ergodicity, sufficiency of explanatory variables, etc. that are quite case dependent. What is wrong is that it is difficult to choose an single extrapolation scheme that works fine in different contexts, without a lot of extra information.

This generic mismatch is clearly illustrated in the Anscombe quartet dataset shown below. The linear regression is also (outside the $x$-coordinate range) an instance of extrapolation. The same line regresses four set of points, with the same standard statistics. However, the underlying models are quite different: the first one is quite standard. The second is a parametric model error (a second or third degree polynomial could be better suited), the third shows a perfect fit except for one value (outlier?), the fourth a lack of smooth relationships (hysteresis?).

However, forecasting can be rectified to some extend. Adding to other answers, a couple of ingredients can help practical extrapolation:

1. You can weight the samples according to their distance (index $n$) to the location $p$ where you want to extrapolate. For instance, use an increasing function $f_p(n)$ (with $p\ge n$), like exponential weighting or smoothing, or sliding windows of samples, to give less importance to older values.
2. You can use several extrapolation models, and combine them or select the best (Combining forecasts, J. Scott Armstrong, 2001). Recently, there have been a number of works on their optimal combination (I may provide references if needed).

Recently, I have been involved in a project for extrapolating values for the communication of simulation subsystems in a real-time environment. The dogma in this domain was that extrapolation may cause instability. We actually realized that combining the two above ingredients was very efficient, without noticeable instability (without a formal proof yet: CHOPtrey: contextual online polynomial extrapolation for enhanced multi-core co-simulation of complex systems, Simulation, 2017). And the extrapolation worked with simple polynomials, with a very low computational burden, most of the operations being computed beforehand and stored in look-up tables.

Finally, as extrapolation suggests funny drawings, the following is the backward effect of linear regression:

Answer 4

Although the fit of a model might be "good", extrapolation beyond the range of the data must be treated skeptically. The reason is that in many cases extrapolation (unfortunately and unavoidably) relies on untestable assumptions about the behaviour of the data beyond their observed support.

When extrapolating one must do two judgement calls: First, from a quantitative perspective, how valid is the model outside the range of the data? Second, from a qualitative perspective, how plausible is a point $x_{out}$ laying outside the observed sample range to be a member of the population we assume for the sample? Because both questions entail a certain degree of ambiguity extrapolation is considered an ambiguous technique too. If you have reasons to accept that these assumptions hold, then extrapolation is usually a valid inferential procedure.

An additional caveat is that many non-parametric estimation techniques do not permit extrapolation natively. This problem is particularly noticeable in the case of spline smoothing where there are no more knots to anchor the fitted spline.

Let me stress that extrapolation is far from evil. For example, numerical methods widely used in Statistics (for example Aitken's delta-squared process and Richardson's Extrapolation) are essentially extrapolation schemes based on the idea that the underlying behaviour of the function analysed for the observed data remains stable across the function's support.

Answer 5

Contrary to other answers, I'd say that there is nothing wrong with extrapolation as far as it is not used in mindless way. First, notice that extrapolation is:

the process of estimating, beyond the original observation range, the value of a variable on the basis of its relationship with another variable.

...so it's very broad term and many different methods ranging from simple linear extrapolation, to linear regression, polynomial regression, or even some advanced time-series forecasting methods fit such definition. In fact, extrapolation, prediction and forecast are closely related. In statistics we often make predictions and forecasts. This is also what the link you refer to says:

We’re taught from day 1 of statistics that extrapolation is a big no-no, but that’s exactly what forecasting is.

Many extrapolation methods are used for making predictions, moreover, often some simple methods work pretty well with small samples, so can be preferred then the complicated ones. The problem is, as noticed in other answers, when you use extrapolation method improperly.

For example, many studies show that the age of sexual initiation decreases over time in western countries. Take a look at a plot below about age of first intercourse in the US. If we blindly used linear regression to predict age of first intercourse we would predict it to go below zero at some number of years (accordingly with first marriage and first birth happening at some time after death)... However, if you needed to make one-year-ahead forecast, then I'd guess that linear regression would lead to pretty accurate short term predictions for the trend.

(source guttmacher.org)

Another great example comes from completely different domain, since it is about "extrapolating" for test done by Microsoft Excel, as shown below (I don't know if this is already fixed or not). I don't know the author of this image, it comes from Giphy.

All models are wrong, extrapolation is also wrong, since it wouldn't enable you to make precise predictions. As other mathematical/statistical tools it will enable you to make approximate predictions. The extent of how accurate they will be depends on quality of the data that you have, using methods adequate for your problem, the assumptions you made while defining your model and many other factors. But this doesn't mean that we can't use such methods. We can, but we need to remember about their limitations and should assess their quality for a given problem.

Answer 6

I quite like the example by Nassim Taleb (which was an adaptation of an earlier example by Bertrand Russell):

Consider a turkey that is fed every day. Every single feeding will firm up the bird's belief that it is the general rule of life to be fed every day by friendly members of the human race "looking out for its best interests," as a politician would say. On the afternoon of the Wednesday before Thanksgiving, something unexpected will happen to the turkey. It will incur a revision of belief.

Some mathematical analogs are the following:

• knowledge of the first few Taylor coefficients of a function does not always guarantee that the succeeding coefficients will follow your presumed pattern.

• knowledge of a differential equation's initial conditions does not always guarantee knowledge of its asymptotic behavior (e.g. Lorenz's equations, sometimes distorted into the so-called "butterfly effect")

Here is a nice MO thread on the matter.

Answer 7

I also remember sitting in a Statistics course, and the professor told us extrapolation was a bad idea. Then during the next class he told us it was a bad idea again; in fact, he said it twice.

I was sick for the rest of the semester, but I was certain I couldn't have missed a lot of material, because by the last week the guy must surely have been doing nothing but telling people again and again how extrapolation was a bad idea.

Strangely enough, I didn't score very high on the exam.

Answer 8

The question is not just statistical; it's also epistemological. Extrapolation is one of the ways we learn about the nature; it's a form of induction.

Let's say we have data for electrical conductivity of a material in a range of temperatures from 0 to 20 Celsius. What can we say about the conductivity at 40 degree Celsius?

It's closely related to small sample inference: what can we say about the entire population from measurements conducted on a small sample? This was started by Gosset at Guinness, who came up with Student t-distributions. Before him statisticians didn't bother to think about small samples assuming that the sample size can always be large. He was at Guinness and had to deal with samples of beer to decide what to do with the entire batch of beer to ship.

So, in practice (business), engineering and science we always have to extrapolate in some ways. It could be extrapolating small samples to larger ones, or from a limited range of input conditions to a wider set of conditions, from what's going on in the accelerator to what happened to a black hole billions of miles away etc. It's especially important in science though, as we really learn by studying the discrepancies between our extrapolation estimates and actual measurements. Often we find new phenomena when the discrepancies are large or consistent.

Hence, I say there is no problem with extrapolation. It's something we have to do every day. It's just difficult.

Answer 9

Extrapolation itself isn't necessarily evil, but it is a process which lends itself to conclusions which are more unreasonable than you arrive at with interpolation.

• Extrapolation is often done to explore values quite far from the sampled region. If I'm sampling 100 values from 0-10, and then extrapolate out just a little bit, merely to 11, my new point is likely 10 times further away from any datapoint than any interpolation could ever get. This means that there's that much more space for a variable to get out of hand (qualitatively). Note that I intentionally chose only a minor extrapolation. It can get far worse
• Extrapolation must be done with curve fits that were intended to do extrapolation. For example, many polynomial fits are very poor for extrapolation because terms which behave well over the sampled range can explode once you leave it. Good extrapolation depends on a "good guess" as to what happens outside of the sampled region. Which brings me to...
• It is often extremely difficult to use extrapolation due to the presence of phase transitions. Many processes which one may wish to extrapolate on have decidedly nonlinear properties which are not sufficiently exposed over the sampled region. Aeronautics around the speed of sound are an excellent example. Many extrapolations from lower speeds fall apart as you reach and exceed the speed of information transfer in the air. This also occurs quite often with soft sciences, where the policy itself can impact the success of the policy. Keynesian economics extrapolated out how the economy would behave with different levels of inflation, and predicted the best possible outcome. Unfortunately, there were second order effects and the result was not economic prosperity, but rather some of the highest inflation rates the US has seen.
• People like extrapolations. Generally speaking, people really want someone to peer into a crystal ball and tell them the future. They will accept surprisingly bad extrapolations simply because it's all the information they have. This may not make extrapolation itself bad, per se, but it is definitely something one should account for when using it.

For the ultimate in extrapolation, consider the Manhattan Project. The physicists there where forced to work with extremely small scale tests before constructing the real thing. They simply didn't have enough Uranium to waste on tests. They did the best they could, and they were smart. However, when the final test occurred, it was decided that each scientist would decide how far away from the blast they wanted to be when it went off. There was substantial differences of opinion as to how far away was "safe" because every scientists knew they were extrapolating quite far from their tests. There was even a non-trivial consideration that they might set the atmosphere on fire with the nuclear bomb, an issue also put to rest with substantial extrapolation!

Answer 10

Lots of good answers here, I just want to try and synthesize what I see as the core of the issue: it is dangerous to extrapolate beyond that data generating process that gave rise to the estimation sample. This is sometimes called a 'structural change'.

Forecasting comes with assumptions, the main one being that the data generating process is (as near as makes no significant difference) the same as the one that generated the sample (except for the rhs variables, whose changes you explicitly account for in the model). If a structural change occurs (i.e. Thanksgiving in Taleb's example), all bets are off.