I'm trying to replicate parts of:

Estrella, A., & Mishkin, F. S. (1998). Predicting U.S. Recessions: Financial Variables as Leading Indicators. Review of Economics and Statistics, 80(1), 45–61.

Here is a link to a freely available working paper version.

As the title suggest, in this paper they compare out-of-sample forecasting accuracy of different financial variable as predictor of recessions using probit models. The procedure of making out-of-sample forecasts is as follows (p. 14):

The out-of-sample results are obtained in the following way. First, a given model is estimated using data from the beginning of the sample up to a particular quarter, say the first quarter of 1970. Then these estimates are used to form projections, say four quarters ahead. In this case, the projection would apply to the first quarter of 1971. After adding one more quarter to the estimation period, the procedure is repeated. That is, data up to the second quarter of 1970 are used to make a projection for the second quarter of 1971. In this way, the procedure mimics what a statistical model would have predicted with the information available at any point in the past. Data that became available subsequent to the prediction date are not used to estimate or to predict recessions.

I was able to replicate this in R (with monthly data instead of quarterly). In any month $t$ the model coefficients are estimated for the model: $$ P(R_{t+k} = 1| X_t) = \Phi\left(\alpha+\beta X_t\right)$$

These coefficients are then used to forecast a probability of recession in $t+k$. Then, one more month is added to the data, the coefficients re-estimated and a new forecast for is made. As such, per forecasting horizon, each estimated model yields one forecast (as single value). I can then plot this as a time series and compare to actual recessions.

So far I think I understand, but what I am unclear on is how they calculate one out-of-sample pseudo-$R^2$ value for all these models (for each forecasting horizon $k$). To me it seems that you could calculate one pseudo-$R^2$ for each model, but how are these aggregated into one number for each forecasting horizon to be able to compare their out-of-sample accuracy?

I tried to understand this answer, but I don't think it is possible to apply this to this situation as I have more than one set of estimated coefficients.

I suppose it is possible to just calculate a regular $R^2$ based on the fitted recension probabilities and the actual recession dummy, but this would not consider the probit form of the model.

What am I missing here?

Note: in this paper they use an alternative pseudo-$R^2$ definition, which I have not seen used in many other papers. I would like to use McFadden's pseudo-$R^2$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.