I'm reading through "An Introduction to Statistical Learning" . In chapter 2, they discuss the reason for estimating a function $f$.
2.1.1 Why Estimate $f$?
There are two main reasons we may wish to estimate f : prediction and inference. We discuss each in turn.
I've read it over a few times, but I'm still partly unclear on the difference between prediction and inference. Could someone provide a (practical) example of the differences?
Inference: Given a set of data you want to infer how the output is generated as a function of the data.
Prediction: Given a new measurement, you want to use an existing data set to build a model that reliably chooses the correct identifier from a set of outcomes.
Inference: You want to find out what the effect of Age, Passenger Class and, Gender has on surviving the Titanic Disaster. You can put up a logistic regression and infer the effect each passenger characteristic has on survival rates.
Prediction: Given some information on a Titanic passenger, you want to choose from the set $\{\text{lives}, \text{dies}\}$ and be correct as often as possible. (See bias-variance tradeoff for prediction in case you wonder how to be correct as often as possible.)
Prediction doesn't revolve around establishing the most accurate relation between the input and the output, accurate prediction cares about putting new observations into the right class as often as possible.
So the 'practical example' crudely boils down to the following difference: Given a set of passenger data for a single passenger the inference approach gives you a probability of surviving, the classifier gives you a choice between lives or dies.
Tuning classifiers is a very interesting and crucial topic in the same way that correctly interpreting p-values and confidence intervals is.
In page 20 of the book, the authors provide a beautiful example which made me understand the difference.
Here's the paragraph from the book : An Introduction to Statistical Learning
" For example, in a real estate setting, one may seek to relate values of homes to inputs such as crime rate, zoning, distance from a river, air quality, schools, income level of community, size of houses, and so forth. In this case one might be interested in how the individual input variables affect the prices—that is, how much extra will a house be worth if it has a view of the river? This is a inference problem. Alternatively, one may simply be interested in predicting the value of a home given its characteristics: is this house under- or over-valued? This is a prediction problem. "
Generally when doing data analysis we imagine that there is some kind of "data generating process" which gives rise to the data, and inference refers to learning about the structure of this process while prediction means being able to actually forecast the data that come from it. Oftentimes the two go together, but not always.
An example where the two go hand in hand would be the simple linear regression model
$$ Y_i = \beta_0 + \beta_1 x_i + \epsilon_i . $$
Inference in this case would mean estimating the parameters of the model $\beta_0$ and $\beta_1$ and our predictions would just be computed from our estimates of these parameters. But there are other types of models where one is able to make sensible predictions, but the model doesn't necessarily lead to meaningful insights about what is happening behind the scenes. Some examples of these kinds of models would be complicated ensemble methods which can lead to good predictions but are sometimes difficult or impossible to understand.
Prediction uses estimated f to forecast into the future. Suppose you observe a variable $y_t$, maybe it's the revenue of the store. You want to make financial plans for your business, and need to forecast the revenue in next quarter. You suspect that the revenue depends on the income of population in this quarter $x_{1,t}$ and the time of the year $x_{2,t}$. So, you posit that it is a function: $$y_t=f(x_{1,t-1},x_{2,t-1})+\varepsilon_t$$
Now, if you get the data on income, say personal disposable income series from BEA, and construct the time of year variable, you may estimate the function f, then plug the latest values of the population income and the time of the year into this function. This will yield the prediction for the next quarter of the revenue of the store.
Inference uses estimated function f to study the impact of the factors on the outcome, and do other things of this nature. In my earlier example you might be interested in how much the season of the year determines the revenue of the store. So, you could look at the partial derivative $\partial f/\partial x_{2t}$ - sensitivity to the season. If f was in fact a linear model then it would be a regression coefficient of the second variable $\beta_2x_{2,t-1}$.
Prediction and inference may use the same estimation procedure to determine f, but they have different requirements to this procedure and incoming data. A well-known case is so called collinearity, whereas your input variables are highly correlated with each other. For instance, you measure weight, height and belly circumference of obese people. It is likely that these variables are strongly correlated, not necessarily linearly though. It happens so that collinearity can be a serious issue for inference, but merely an annoyance to prediction. The reason is that when predictors $x$ are correlated it's harder to separate the impact of predictor from the impact of other predictors. For prediction this doesn't matter, all you care is the quality of the forecast.
You are not alone here. After reading answers, I am not confused anymore - not because I understand the difference, but because I understand it is in the eye of the beholder and verbally induced. I am sure now those two terms are political definitions rather than scientific ones. Take for example the explanation from the book, the one that colleges tried to use as a good one: "how much extra will a house be worth if it has a view of the river? This is a inference problem." From my point of view, this is absolutely a prediction problem. You are civil construction company owner, and you want to choose the best ground for building next set of houses. You have to choose between two location in the same town, one near the river, the next near the train station. You want to predict the prices for both locations. Or you want to infer. You are going to apply the exact methods of statistics, but you name the process. :)
Given a data set of $n=100$ observations, $k=50$ independent variables $x_i$, and one dependent variable $y$, inference answers questions such as:
Both of these questions are questions about the parameters in the “true model” that generated the data.
Prediction answers a much simpler question:
This question does not ask anything about the parameters in the true model. Nor does it require the existence of a “true model”. Prediction simply involves a plug-and-chug to generate a value $\hat{y}$ that is ideally close to $y$.
Imagine, you are a medical doctor on an intensive care unit. You have a patient with a strong fever and a given number of blood cells and a given body weight and a hundred different data and you want to predict, if he or she is going to survive. If yes, he is going to conceal that story about his other kid to his wife, if not, it is important for him do reveal it, while he can.
The doctor can do this prediction based on the data of former patients he had at his unit. Based on his software knowledge, he can predict using either a generalized linear regression (glm) or via a neural net (nn).
There are far to many correlated parameters for the glm so to get to a result, the doctor will have to make assumptions (linearity etc.) and decisions about which parameters are likely to have an influence. The glm will reward him with a t-test of significance for each of his parameters so he might gather strong evidence, that gender and fever have a significant influence, body weight not necessarily so.
The neural net will swallow and digest all information that there is in the sample of former patients. It will not care, whether predictors are correlated and it will not reveal that much information, on whether the influence of body weight seems to be important only in the sample at hand or in general (at least not at the level of expertise that the doctor has to offer). It will just compute a result.
What method to choose depends on the angle from which you look on the problem: As a patient, I would prefer the neural net which uses all available data for a best guess on what will happen to me without strong and obviously wrong assumptions like linearity. As the doctor, who wants to present some data in a journal, he needs p-values. Medicine is very conservative: they are going to ask for p-values. So the doctor wants to report, that in such a situation, gender has a significant influence. For the patient, that does not matter, just use whatever influence the sample suggests to be most likely.
In this example, the patient wants prediction, the scientist-side of the doctor wants inference. Mostly, when you want to understand a system, then inference is good. If you need to make a decision where you cannot understand the system, prediction will have to suffice.
I know many answers have been posted already, but for those of you who don't read the book (Introduction to Statistical Learning), here's three exercises found in the second chapter. See if you can solve them, they helped me quite a bit to understand the difference between inference and prediction.
Explain whether each scenario is a classification or regression problem, and indicate whether we are most interested in inference or prediction.
We collect a set of data on the top 500 firms in the US. For each firm we record profit, number of employees, industry and the CEO salary. We are interested in understanding which factors affect CEO salary.
We are considering launching a new product and wish to know whether it will be a success or a failure. We collect data on 20 similar products that were previously launched. For each product we have recorded whether it was a success or failure, price charged for the product, marketing budget, competition price, and ten other variables.
We are interesting in predicting the % change in the US dollar in relation to the weekly changes in the world stock markets. Hence we collect weekly data for all of 2012. For each week we record the % change in the dollar, the % change in the US market, the % change in the British market, and the % change in the German market.
If you want the answers, they can be found here. Note that the exercise above is number 2.
There's good research showing that a strong predictor of whether borrowers will repay their loans is whether they use felt to protect their floors from being scratched by furniture legs. This "felt" variable will be a distinct aid to a predictive model where the outcome is repay vs. default. However, if lenders want to gain greater leverage over this outcome, they will be remiss in thinking they can do so by distributing felt as widely as they can.
"How likely is this borrower to repay?" is a prediction problem; "How can I influence the result?" is a causal inference problem.
y = f(x) then
prediction(what is the value of Y with a given value of x: if specific value of x what could be the value of Y
inference(how y changes with change in x) : what could be the affect on Y if x changes
Prediction example : suppose y represent the salary of a person then if we provide input such as years of experience, degree as input variables then our function predicts the salary of the employee.
Inference example : suppose cost of living changes then how much is the change in salary
