Is a Chow test appropriate for this intervention analysis?

Question

I am fairly new to R was wondering if this approach was appropriate. Sorry if I forget anything in my question.

I have long-form panel data for 15 years (2001-2015) for hundreds of counties and corresponding BEA linecode data for each of these counties (i.e., personal income, employment, etc).

For each county, I have known intervention times (some have multiple) that occur sometime between 2001-2015. I need to test whether or not the intervention had a significant effect after it was implemented in the simplest way possible.

After researching intervention analysis extensively, I thought that I would try using ARIMA to forecast what the data would look like after the intervention and determine whether the difference between the forecasted values and the actual values were statistically significant. If I had only a few counties, I could manually review each one and determine the best arima model, whether or not there was autocorrelation, account for stationary/non stationary data, however, because I have so many counties to analyze, I abandoned the forecasting approach.

I am now using the Chow test (see link 1 below) because I think I can apply this to all my counties? I settled on this method because I know the intervention periods and if I sort each county by when the intervention occurred, I can run a loop to analyze all counties that have an intervention in point=8, (2008), then point=9 for example. I will provide sample data below.

1. CHOW TEST EXAMPLE: Comparing slopes in a time series following an intervention

2. Using Wald test to compare: Method for quantifying intervention effect in time series

DATA: I first started by using a subset of one county (Essex County in NJ) and one line code (linecode 10 of table ca30, personal income). This particular county has an intervention in 2012.

X <-c(2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015)

Y <-c(43366097,42933848,43455138,44675533,44160176,45388263,45985239,45641473,45016810,44807495,45632509,45136399,45120870,46490467,48492136)

I used the Chow test

sctest(Y~X,type="Chow",point=12)

and the summary gave me

Chow test

data:  Y ~ X
F = 5.1536, p-value = 0.02635

Signifying that in 2012, there was a structural change in the data.

Would using this method be appropriate for the rest of my data? Keep in mind I wouldn't be able to manually examine every county BEA combo.

My next step, if this test is appropriate, would be to write some sort of loop that runs through every county, each linecode of importance, determines the year of intervention, runs the chow test for that county/linecode combo and specific year.

I think I would need to ask another question about the loop because I have no experience writing loops but my thought process is that the loop would need to be subsetting each county/linecode combo to be able to isolate the chow test for that specific county/linecode.

Answer 1

The Chow test compares two sets of coefficients for a pre-specified particular model in order to asses a significant collective change in the coefficients from group 1 to group 2. It appears that you are assuming that a simple time trend model is an adequate representation of the data within each of your two sets (2001-2011 and 2012-2015) . I would think not . A more viable model alternative is that there was a level shift at period 2006 with an anomaly at period 2015.

This would conclude that there is no evidence of an assignable cause/effect starting at period 2012. By the way , I spend a lot of time in New jersey !

My conclusion is that the Chow Test ( and I am a big fan of Gregory having incorporated his test into AUTOBOX to find break-points in a set of parameters ) is not applicable here .

I took the 15 data points and used AUTOBOX to construct a dummy variable X having values 0,0,0,0,0,0,0,0,0,0,0,1,1,1,1 and estimated an OLS model with stats here . AUTOBOX's .0225 is your.02635 with a slight difference due to estimation convergence . This is very simply a test of the hypothesis " is there a level shift at time period 12 given no outliers , no ARIMA memory and no prior level shifts , no trends et al." That is too many caveats for me .... . I can't imagine who suggested this test to you in order to answer the question "did anything unusual happen systematically starting at 2012 to change the level of the expected value ?"