2
$\begingroup$

I am trying to fit a model of the form

$$y=f(x)+u$$

where $x$ is a vector variable and $u$ an independent random noise. I tried to fit the unknown function $f$ by different machine learning methods (SVM, MARS, etc.)

I split my data into training and validation sets. After fitting the model on the training set, I test it on a validation set, and I plot $y$-realized versus $y$-predicted, and do a linear regression on that scatter plot. I obtain a very small intercept but a coefficient of $0.5$, while I was naively expecting a coefficient near to 1.

Does this regression coefficient tell me something about the quality of fit of my original model or about the original model itself?

$\endgroup$
2
$\begingroup$

Does this regression coefficient tell me something about the quality of fit of my original model or about the original model itself?

Yes, it does. Indeed, a good forecast should be associated with a zero intercept and a unit slope in your setting. Deviations from that indicate forecast suboptimality. The idea is formalized by the Mincer-Zarnowitz test. Given forecasts $\hat y_i$ and realized values $y_i$, the regression

$$ y_i = \beta_0 + \beta_1 \hat y_i + \varepsilon_i $$

should ideally (optimally) have zero intercept and unit slope, i.e. $\beta_0=0$ and $\beta_1=1$, which constitutes a testable hypothesis (the Mincer-Zarnowitz test); see Diebold "Forecasting in Economics, Business, Finance and Beyond", p. 337, for an introduction (the textbook is regularly updated so the page number may change over time; just search the pdf for "Mincer" and you will find it).

However, note that a zero intercept and a unit slope do not completely determine the goodness of your forecast. $\beta_0=0$ and $\beta_1=1$ indicates good accuracy but does not say anything about precision. You could have $\beta_0=0$ and $\beta_1=1$ combined with very high error variance $\text{Var}(\varepsilon_i)$ so that mean squared forecast error would still be high and hence the forecast not very satisfactory.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.