0
$\begingroup$

I am building a model to forecast housing variables using vector autoregression. I am encountering spurious results. My forecasted values change dramatically depending in which month the historical data end. I have several questions and appreciate any advice you could give. My questions are mostly conceptual and statistical. First I describe the purpose; then the data; a summary of the model, including a few lines of R code; then the problem, and finally I discuss several issues that could be causing the problem. My focus here is not on coding, but provide this information to clarify my approach.

Purpose

The main purpose of the project is both the monthly and yearly forecasts, forecasting the balance of the current year plus two years. The forecast will be updated periodically. In my discussion here, I am mostly assuming that the last historical data available are as of December 2015 and am mostly interested in the 2016 prediction.

Data and Transformations

My data start in 2000, are monthly and mostly non-seasonal.(*) I standardize sales and housing starts by dividing by the number of households. I also take the first difference (or the log of the first difference) of all variables to obtain stationary transformations of the variables. Data mostly come from government sources. I have both endogenous and exogenous data variables, which are listed and described below. I also have future values of the exogenous data, forecasted from an entirely other model and take them as given.

endogenous variables:

d_sfhs_hh = the first difference (d) of single family home sales (sfhs) divided by the number of households (hh)

d_sfhp = the first difference (d) of single family home median prices (sfhp)

d_starts = the first difference (d) of housing starts (starts)

exogenous variables:

dl_interest_rate = first difference (dl) of the log of interest rate (interest_rate)

d_gdp = the first difference (d) of gross domestic product (gdp)

d_med_hh_inc = the first difference (d) of median household income (med_hh_inc)

d_unmr = the first difference (d) of the unemployment rate (unmr)

Once I forecast my monthly values, I reverse the transformations (reverse the differencing and multiply out the number of households). I then aggregate these monthly forecasts to produce the yearly forecast.

(*) Households use a "seasonally adjusted" variant, but since there is little seasonality in housing, the distinction is trivial. GDP is a mathematical transformation of quarterly data.

Core Model Code

Here are a few lines of modeling code. Obviously there are many other elements of code that I use to acquire, test, transform, visualize, model, reverse the transformations and report the data. But since this is not the forum for discussing R coding, I simply provide a few lines of core coding to clarify my modeling approach.

I combine the data as:

endog <- cbind(d_sfhs_hh, d_sfhp, d_starts)
exog <- cbind(dl_interest_rate, d_gdp, d_med_hh_inc, d_unmr)
exog_ <- cbind(dl_interest_rate_, d_gdp_, d_med_hh_inc_, d_unmr_)   

exog_ are the future estimates of the exogenous variables, which I supply, since these are not predicted through VAR.

prediction <- predict(VAR(endog, p = 12, season=12, type = 'both', exogen = exog), dumvar = exog_, n.ahead = 24, ci = 0.95)

The Problem

The principle problem that I am having is that my yearly forecasts vary greatly depending on which month I forecast from. I can also arbitrarily end the historical series at any point. For example, if I forecast using historical data through December of 2015, the VAR model forecasts a 7 percent decline in sales for 2016. If, however, I forecast using historical data through October of 2015, the model forecasts a 6 percent increase in sales for 2016. From what we know at the time of this post (May 23, 2016), 2016 sales are up from 2015 and the industry expects a 4 to 5 growth percent for 2016 compared to 2015.

While we expect large seasonal swings in monthly sales with a low during winter and high during spring, this seasonal variation should not affect yearly aggregations. It should make little difference whether our historical data stopped in October or December, except for 1) an increase in the margin of error because the October historical data has to forecast an extra two months and 2) an impact if there were some exogenous macroeconomic shock that took place during November or December. There were no significant shocks during these two months, so we expect the December forecast to be in line with the October forecast.

When I forecast one time period out (one month) the model forecast error is only about 2 percent. When I visually examine a 24 to 36 period (2 to 3 years) forecast, the seasonal pattern appears correct. However, aggregating these values over a year, can be very low or very high, suggesting systematic bias, perhaps auto correlation. The odd thing is that the bias can be either positive or negative, depending upon which month the historical data stop.

Potential Issues

I can identify several potential areas that could be causing the problem, but don't have enough VAR experience to assess these and seek your advice.

First, should I be using two separate models using non-seasonal data for monthly forecasts and seasonal data for yearly forecasts? This solution makes little sense to me, but suggest it any way because I am stumped.

Second, I could be setting the p value (the maximum number of lags to include) incorrectly. AIC suggests a value of p=12, but I have also tried p=1 and several other values and while the choice of p affects the forecasts, I was hoping the correct value would stabilize my historical ending month problem. That said, I have my doubts that this is the cause.

While I haven't tried all combinations of p values 1 to 12 for a variety of historical data ending months, I have tried a few combinations, such as p=1, p=2 and p=12 for October and December historical data ending months. I haven't yet found the type of convergence that I seek.

At first I thought p=12 conceptually made sense because of the seasonal variation, but then I realized that a variable is seasonally correlated mostly with itself, i.e., May is always a good month for home sales and may be confusing seasonality with lagged causality. But VAR is also modeling the impact of lagged values of each variable against the others, which suggests looking at granger causality.

The granger causality test examines the impact of one variable on the future value of another variable. What I found is, for example, the price (d_sfhp) to housing starts (d_starts) granger test was most significant with 10 and 11 months, although not 12 months. Conceptually 10 to 11 months makes sense, compared to 2 or 3 months. A significant hike in home sale prices is not going to affect starts the next month. It can take almost a year from the point that the developer decides to build until she or he gets the permits and schedules the start of excavation. In this case, the lag effect is not weather related or other yearly seasonality, but the average administrative time it takes between the decision to build and the first shovel touching the ground. The US Bureau of the Census defines a housing start when excavation begins.

Conceptually, each pair of values could have a different lag time for granger causality. For example, I would expect significant changes in the interest rate to have a much shorter lag time. Indeed, while granger causality between the interest rate (interest_rate) and sales (sfhs) was significant for most lags from 1 to over 12 months, the most significant was a 1 month lag. Note, however, that granger causality was much less significant for the log of the difference of interest rates (dl_interest_rate) and the difference of sales (d_sfhs). So, in the real world, causality is lagged by a different time period for each between pair of variables. In fact there could even be a negative lag. I discussed this with someone in real estate, who believed that consumers will respond to anticipated changes in the interest rate. For example, the financial news suggests that the Fed will raise the interest rate next month. Buyers, sitting on the fence will act (or delay) quickly in response to something that they believe will happen, which is a negative lag or just happened, which is a small positive lag.

VAR, at least in R implementation, only permits one value for p. Is that because it is a maximum lag? Hence in this discussion, probably a value of 10 to 12 for p, would be best because it captures the impact of price on starts, even though many degrees of freedom are lost on the impact of the interest rate on sales, since most of this is within the first month and the shorter lags for starts would not add power, but would cost degrees of freedom. I guess this suggests the trade off between the ease and flexibility of a VAR model and the parsimony that can be achieved with a structural model.

So, do you think the choice of p could be causing the historical ending month instability? I suspect not, but seek your response.

Third, in R, does p affect the exogenous variables. In other words with a p of 5, would lags 1 to 5 of the exogenous variables enter the equation, or does p only affect the endogenous variables and only the contemporary value of the exogenous variable at t, not lagged at t-1, t-2, ... enter the equation? If the latter is the case, then one could achieve some parsimony by explicitly setting the lagged value of the exogenous variable. In my case, I would enter lag(dl_interest_rate), because the grangertest was highly significant for order=1. This too, I doubt is causing my issue one way or another, but seek your comments.

Fourth, I wonder if the problem is the way that I have transformed the data. I chose these transformations because they achieve stationarity. Perhaps, instead there is cointegration and I should try another approach. Any advice on whether my approach here is causing the historical ending period issue?

Fifth, I could be missing some important variables, but I doubt this would cause the issue at hand. Any thoughts?

Sixth, I could be coding something incorrectly. While this is not the forum for detailed discussion of R coding, if your wisdom is that none of the above or other conceptual or statistical issues is the problem and it appears that the only explanation could be coding, then please point me in this direction.

$\endgroup$
  • $\begingroup$ The order of operations should be $\Delta\log(x)$ rather than $\log(\Delta x)$, which is the impression from the text. Of course, in the application you must have the right order as otherwise you would immediately be getting errors from taking logs of negative values. $\endgroup$ – Richard Hardy May 24 '16 at 19:55
  • $\begingroup$ Third, in R, does p affect the exogenous variables. No. But you can always double check by looking at the model output. Your post is very long, which may put off some users. I understand you are trying to be as explicit as possible to make everything clear and nice, though. My initial guess of your problem is that you have too many variables in the model, making it hypersensitive to the data, which harms forecasts. Any random wiggles are perhaps interpreted as genuine signal when you have 12 lags of multiple variables. Does lag order 1 or 2 still perform as poorly in forecasting? $\endgroup$ – Richard Hardy May 24 '16 at 20:01
  • $\begingroup$ I have mostly run the model with p=1. I have also tried dropping the exogenous variables, leaving only three endogenous variables at p=1, but this does not resolve the issue. The problem is likely elsewhere. I reviewed my actual code and yes, it is the difference of the log. I have made the edit in my post. Would you suggest I edit this post to shorten it or just keep this in mind for the next time? $\endgroup$ – Joe May 24 '16 at 20:47
  • $\begingroup$ It's your choice. There is nothing wrong with it, I am just thinking of how to engage more users. $\endgroup$ – Richard Hardy May 24 '16 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.