29
$\begingroup$

I'm reading through "An Introduction to Statistical Learning" . In chapter 2, they discuss the reason for estimating a function $f$.

2.1.1 Why Estimate $f$?

There are two main reasons we may wish to estimate f : prediction and inference. We discuss each in turn.

I've read it over a few times, but I'm still partly unclear on the difference between prediction and inference. Could someone provide a (practical) example of the differences?

$\endgroup$
  • 4
    $\begingroup$ The authors of An Introduction to Statistical Learning have done us a disservice here. One makes inferences to make predictions just as one makes inferences to understand causes and effects. Merriam-webster.com defines "infer" principally as "to derive as a conclusion from facts or premises." That covers both causal and predictive reasoning. It's misleading, confusing, and against long-term standard usage to narrow down the definition of "infer" to include only matters of causality. So: let's contrast predictive inference with causal inference. $\endgroup$ – rolando2 Aug 8 '18 at 12:40
  • 1
    $\begingroup$ @rolando2: Then I guess we also need descriptive inference, that is, inference about parameters in descriptive (non-causal) models. $\endgroup$ – kjetil b halvorsen Nov 1 '18 at 10:16
24
$\begingroup$

Inference: Given a set of data you want to infer how the output is generated as a function of the data.

Prediction: Given a new measurement, you want to use an existing data set to build a model that reliably chooses the correct identifier from a set of outcomes.


Inference: You want to find out what the effect of Age, Passenger Class and, Gender has on surviving the Titanic Disaster. You can put up a logistic regression and infer the effect each passenger characteristic has on survival rates.

Prediction: Given some information on a Titanic passenger, you want to choose from the set $\{\text{lives}, \text{dies}\}$ and be correct as often as possible. (See bias-variance tradeoff for prediction in case you wonder how to be correct as often as possible.)


Prediction doesn't revolve around establishing the most accurate relation between the input and the output, accurate prediction cares about putting new observations into the right class as often as possible.

So the 'practical example' crudely boils down to the following difference: Given a set of passenger data for a single passenger the inference approach gives you a probability of surviving, the classifier gives you a choice between lives or dies.

Tuning classifiers is a very interesting and crucial topic in the same way that correctly interpreting p-values and confidence intervals is.

$\endgroup$
  • 1
    $\begingroup$ Good answer. But when some people hear the word "inference," they think "causal inference." You might wish to say something about that as well, even though (or perhaps especially because) ISLR isn't focused on it. $\endgroup$ – generic_user Nov 3 '16 at 15:32
  • 1
    $\begingroup$ I think in the deep learning world inference is pretty much the same as prediction eg. blogs.nvidia.com/blog/2016/08/22/… $\endgroup$ – user1893354 Dec 12 '17 at 0:34
  • 1
    $\begingroup$ This seems like a perfectly good answer to me. $\endgroup$ – gung Jan 19 at 17:54
  • 2
    $\begingroup$ I believe this answer is wrong, specifically "the inference approach gives you a probability of surviving, the classifier gives you a choice between lives or dies", even more specifically the first part. Calculating the probability that a passenger survives is a prediction, specifically a probabilistic one. In the continuous case it would be a density forecast. We can then threshold this predicted probability to get a hard classification, yes. ... $\endgroup$ – Stephan Kolassa Jan 20 at 7:00
  • 3
    $\begingroup$ ... Whereas prediction is about predicting the outcomes, inference is about understanding the relationship of the inputs to the outcome: which input has such a relationship, and how can we distinguish a "true" relationship from random covariation (which is where p values come in)? Can someone explain to me where I am mistaken? $\endgroup$ – Stephan Kolassa Jan 20 at 7:02
10
$\begingroup$

Generally when doing data analysis we imagine that there is some kind of "data generating process" which gives rise to the data, and inference refers to learning about the structure of this process while prediction means being able to actually forecast the data that come from it. Oftentimes the two go together, but not always.

An example where the two go hand in hand would be the simple linear regression model

$$ Y_i = \beta_0 + \beta_1 x_i + \epsilon_i . $$

Inference in this case would mean estimating the parameters of the model $\beta_0$ and $\beta_1$ and our predictions would just be computed from our estimates of these parameters. But there are other types of models where one is able to make sensible predictions, but the model doesn't necessarily lead to meaningful insights about what is happening behind the scenes. Some examples of these kinds of models would be complicated ensemble methods which can lead to good predictions but are sometimes difficult or impossible to understand.

$\endgroup$
  • 2
    $\begingroup$ "but the model doesn't necessarily lead to meaningful insights about what is happening behind the scenes." The term "black box" comes to mind. :) $\endgroup$ – Alexis Aug 8 '18 at 18:02
  • $\begingroup$ or multi-layer neural nets $\endgroup$ – Shihab Shahriar Dec 19 '18 at 16:20
  • $\begingroup$ "But there are other types of models where one is able to make sensible predictions, but the model doesn't necessarily lead to meaningful insights about what is happening behind the scenes.", who cares? The inference problem, as you defined it, is still the one of determining the parameters of the model. I don't understand why you start this sentence with "But" and why you even wrote this sentence, with respect to your definition of inference and prediction. $\endgroup$ – nbro Jan 19 at 12:16
  • 1
    $\begingroup$ The use of "but" is because both the description of the preceding & following models are correct, however the following model is distinct from the nature of the former in ways that you wouldn't have expected until you grasp the distinction between prediction & inference that is being illustrated. I'm already familiar w/ the distinction, nonetheless, I find this example insightful & helpful. +1 $\endgroup$ – gung Jan 19 at 18:00
6
$\begingroup$

In page 20 of the book, the authors provide a beautiful example which made me understand the difference.

Here's the paragraph from the book : An Introduction to Statistical Learning

" For example, in a real estate setting, one may seek to relate values of homes to inputs such as crime rate, zoning, distance from a river, air quality, schools, income level of community, size of houses, and so forth. In this case one might be interested in how the individual input variables affect the prices—that is, how much extra will a house be worth if it has a view of the river? This is a inference problem. Alternatively, one may simply be interested in predicting the value of a home given its characteristics: is this house under- or over-valued? This is a prediction problem. "

$\endgroup$
3
$\begingroup$

Prediction uses estimated f to forecast into the future. Suppose you observe a variable $y_t$, maybe it's the revenue of the store. You want to make financial plans for your business, and need to forecast the revenue in next quarter. You suspect that the revenue depends on the income of population in this quarter $x_{1,t}$ and the time of the year $x_{2,t}$. So, you posit that it is a function: $$y_t=f(x_{1,t-1},x_{2,t-1})+\varepsilon_t$$

Now, if you get the data on income, say personal disposable income series from BEA, and construct the time of year variable, you may estimate the function f, then plug the latest values of the population income and the time of the year into this function. This will yield the prediction for the next quarter of the revenue of the store.

Inference uses estimated function f to study the impact of the factors on the outcome, and do other things of this nature. In my earlier example you might be interested in how much the season of the year determines the revenue of the store. So, you could look at the partial derivative $\partial f/\partial x_{2t}$ - sensitivity to the season. If f was in fact a linear model then it would be a regression coefficient of the second variable $\beta_2x_{2,t-1}$.

Prediction and inference may use the same estimation procedure to determine f, but they have different requirements to this procedure and incoming data. A well-known case is so called collinearity, whereas your input variables are highly correlated with each other. For instance, you measure weight, height and belly circumference of obese people. It is likely that these variables are strongly correlated, not necessarily linearly though. It happens so that collinearity can be a serious issue for inference, but merely an annoyance to prediction. The reason is that when predictors $x$ are correlated it's harder to separate the impact of predictor from the impact of other predictors. For prediction this doesn't matter, all you care is the quality of the forecast.

$\endgroup$
2
$\begingroup$

Imagine, you are a medical doctor on an intensive care unit. You have a patient with a strong fever and a given number of blood cells and a given body weight and a hundred different data and you want to predict, if he or she is going to survive. If yes, he is going to conceal that story about his other kid to his wife, if not, it is important for him do reveal it, while he can.

The doctor can do this prediction based on the data of former patients he had at his unit. Based on his software knowledge, he can predict using either a generalized linear regression (glm) or via a neural net (nn).

1. Generalized Linear Model

There are far to many correlated parameters for the glm so to get to a result, the doctor will have to make assumptions (linearity etc.) and decisions about which parameters are likely to have an influence. The glm will reward him with a t-test of significance for each of his parameters so he might gather strong evidence, that gender and fever have a significant influence, body weight not necessarily so.

2. Neural net

The neural net will swallow and digest all information that there is in the sample of former patients. It will not care, whether predictors are correlated and it will not reveal that much information, on whether the influence of body weight seems to be important only in the sample at hand or in general (at least not at the level of expertise that the doctor has to offer). It will just compute a result.

What's better

What method to choose depends on the angle from which you look on the problem: As a patient, I would prefer the neural net which uses all available data for a best guess on what will happen to me without strong and obviously wrong assumptions like linearity. As the doctor, who wants to present some data in a journal, he needs p-values. Medicine is very conservative: they are going to ask for p-values. So the doctor wants to report, that in such a situation, gender has a significant influence. For the patient, that does not matter, just use whatever influence the sample suggests to be most likely.

In this example, the patient wants prediction, the scientist-side of the doctor wants inference. Mostly, when you want to understand a system, then inference is good. If you need to make a decision where you cannot understand the system, prediction will have to suffice.

$\endgroup$
  • 1
    $\begingroup$ "As a patient, I would prefer the neural net..." You seem to ignore the fact that large amounts of data can be extremely hard to come by in clinical science. Data sets of just several observations per group are not uncommon due to security, privacy and ethical concerns. If you can make defensible assumptions on the data-generating process, you can make far more efficient use of the data. $\endgroup$ – Frans Rodenburg Nov 1 '18 at 9:59
  • $\begingroup$ This was supposed to be a hypothetical scenario in which one can easily relate to why the seemingly same problem can prompt inference questions and prediction problems and why they are not the same. I was not proposing methods to actually predict patient survival chances and yes, I am very well aware of how hard it can be to obtain reliable clinical data of noticeable size. IMHO good assumptions/knowledge of the data-generating process will help in prediction as well as in inference, thus do not much in distinguishing both. $\endgroup$ – Bernhard Nov 1 '18 at 16:14
1
$\begingroup$

You are not alone here. After reading answers, I am not confused anymore - not because I understand the difference, but because I understand it is in the eye of the beholder and verbally induced. I am sure now those two terms are political definitions rather than scientific ones. Take for example the explanation from the book, the one that colleges tried to use as a good one: "how much extra will a house be worth if it has a view of the river? This is a inference problem." From my point of view, this is absolutely a prediction problem. You are civil construction company owner, and you want to choose the best ground for building next set of houses. You have to choose between two location in the same town, one near the river, the next near the train station. You want to predict the prices for both locations. Or you want to infer. You are going to apply the exact methods of statistics, but you name the process. :)

$\endgroup$
  • $\begingroup$ The idea that a view of the river will necessitate a certain price is a causal interpretation. Prediction is agnostic about causality: I can predict effects from causes, causes from effects, or 1 effect from another effect w/ similar causes. Consider my friend Billy, who is 5'10" & who has an identical twin, Bobby, that I have never met. Nonetheless, I can predict Bobby is 5'10", but if I make Billy taller by giving him elevator shoes, I can't necessarily predict Billy will likewise be taller. $\endgroup$ – gung Jan 19 at 17:54
  • $\begingroup$ The example you cite is simply a poor example, that is why it is confusing. There is definitely more to the difference between inference and prediciton than "politics". $\endgroup$ – Richard Hardy May 11 at 16:22
1
$\begingroup$

There's good research showing that a strong predictor of whether borrowers will repay their loans is whether they use felt to protect their floors from being scratched by furniture legs. This "felt" variable will be a distinct aid to a predictive model where the outcome is repay vs. default. However, if lenders want to gain greater leverage over this outcome, they will be remiss in thinking they can do so by distributing felt as widely as they can.

"How likely is this borrower to repay?" is a prediction problem; "How can I influence the result?" is a causal inference problem.

$\endgroup$
-1
$\begingroup$

y = f(x) then

prediction(what is the value of Y with a given value of x: if specific value of x what could be the value of Y

inference(how y changes with change in x) : what could be the affect on Y if x changes

Prediction example : suppose y represent the salary of a person then if we provide input such as years of experience, degree as input variables then our function predicts the salary of the employee.

Inference example : suppose cost of living changes then how much is the change in salary

$\endgroup$
  • $\begingroup$ I don't see why this answer merits two downvotes. $\endgroup$ – gung Jan 19 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.