# Calculating pseudo-$R^2$ for out-of-sample probit model forecasts

I'm trying to replicate parts of:

Estrella, A., & Mishkin, F. S. (1998). Predicting U.S. Recessions: Financial Variables as Leading Indicators. Review of Economics and Statistics, 80(1), 45–61.

Here is a link to a freely available working paper version.

As the title suggest, in this paper they compare out-of-sample forecasting accuracy of different financial variable as predictor of recessions using probit models. The procedure of making out-of-sample forecasts is as follows (p. 14):

The out-of-sample results are obtained in the following way. First, a given model is estimated using data from the beginning of the sample up to a particular quarter, say the first quarter of 1970. Then these estimates are used to form projections, say four quarters ahead. In this case, the projection would apply to the first quarter of 1971. After adding one more quarter to the estimation period, the procedure is repeated. That is, data up to the second quarter of 1970 are used to make a projection for the second quarter of 1971. In this way, the procedure mimics what a statistical model would have predicted with the information available at any point in the past. Data that became available subsequent to the prediction date are not used to estimate or to predict recessions.

I was able to replicate this in R (with monthly data instead of quarterly). In any month $$t$$ the model coefficients are estimated for the model: $$P(R_{t+k} = 1| X_t) = \Phi\left(\alpha+\beta X_t\right)$$

These coefficients are then used to forecast a probability of recession in $$t+k$$. Then, one more month is added to the data, the coefficients re-estimated and a new forecast for is made. As such, per forecasting horizon, each estimated model yields one forecast (as single value). I can then plot this as a time series and compare to actual recessions.

So far I think I understand, but what I am unclear on is how they calculate one out-of-sample pseudo-$$R^2$$ value for all these models (for each forecasting horizon $$k$$). To me it seems that you could calculate one pseudo-$$R^2$$ for each model, but how are these aggregated into one number for each forecasting horizon to be able to compare their out-of-sample accuracy?

I tried to understand this answer, but I don't think it is possible to apply this to this situation as I have more than one set of estimated coefficients.

I suppose it is possible to just calculate a regular $$R^2$$ based on the fitted recension probabilities and the actual recession dummy, but this would not consider the probit form of the model.

What am I missing here?

Note: in this paper they use an alternative pseudo-$$R^2$$ definition, which I have not seen used in many other papers. I would like to use McFadden's pseudo-$$R^2$$

To me, $$R^2$$-style measures compare how your model performs to the performance of a naïve baseline model. In the simple letting of linear regression, that baseline model for predicting conditional expected values is the marginal expected value, which is why the $$\bar y$$ appears in the denominator of this way of writing $$R^2$$.

$$R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right)$$

The predictions from your model are the $$\hat y_i$$. The predictions from the baseline model are $$\bar y$$ every time.

You have time-dependent data, so I would take a somewhat different stance. If you have only a short period of time before when you must make a forecast for a year in the future, you predict the mean from that short period of time. If you have a longer period of time on which you can base your forecast, your baseline model is the mean of that entire period of time. The baseline is dynamic, in that you keep updating it, but this does feel like it is within the spirit of my usual take on $$R^2$$.

Since you want to mimic McFadden's $$R^2$$, you would use the log loss incurred by your model (numerator) and the log loss incurred by this dynamic baseline model (denominator).

$$R^2_{you} =1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left[ y_i\log(\hat y_{i,\text{model}}) + (1-y_i)\log(1-\hat y_{i,\text{model}}) \right] }{ \overset{N}{\underset{i=1}{\sum}}\left[ y_i\log(\hat y_{i,\text{model}}) + (1-y_i)\log(1-\hat y_{i,\text{dynamic baseline}}) \right] }\right)$$

This seems consistent with the article you cite.

In this way, the procedure mimics what a statistical model would have predicted with the information available at any point in the past.

This link discusses Campbell and Thompson (2008) that seems to use this same approach of using a dynamic baseline model. Even if they are not using McFadden's $$R^2$$, I think you can cite them as inspiration, and you have more evidence that people use this kind of dynamic $$R^2$$.

Referring to this as "the" $$R^2$$ seems misleading, since the calculation really is different from the usual ways, but it certainly strikes me as within the spirit of the usual $$R^2$$ being interpreted as a comparison between model performance and the performance of a baseline model.

REFERENCE

Campbell, John Y., and Samuel B. Thompson. "Predicting excess stock returns out of sample: Can anything beat the historical average?." The Review of Financial Studies 21.4 (2008): 1509-1531.