# How to assess effect of intervention in one state versus another using annual death rate data?

I am a beginner in statistics. I have these unpublished data (cases and deaths) of a disease for 7 years (2004-2010) from 2 neighbouring states. The study was started in 2004. State 1 and State 2 received different treatments. The death rate is high in one State. I want to prove that treatment given in State 1 is superior compared to that given in State 2. Time series was not accepted because of many possible confounding variables. I have SPSS and Comprehensive Meta-analysis software. Please guide me.

1. What is the best design? Prospective cohort study or Case-control study?
2. What is the best effect measure (Odds ratio, risk ratio, any other)?
3. What is the best statistical test (?Chi square etc)
4. Can I use meta-analysis in this case?
5. Any other information you think that would be of use.

State 1         State 2
Cases   Deaths  Cases   Deaths
2004    1125    5   2024    254
2005    1213    5   1978    209
2006    1003    4   2294    217
2007    1425    6   2312    249
2008    1172    4   1528    197
2009    1092    3   1683    204
2010    1316    4   2024    218

• Before you get into the statistical side of things, you need to wrestle with the design side--and validity. How do we know these states didn't have radically different rates of the disease before either of the two treatments were begun? There's a conflict between your desire to "prove" (strong word!) the superiority of State 1's treatment and your acknowledgment that there are confounding variables present. Mar 12, 2011 at 3:36
• One designs an experiment before collecting data, not after!
– whuber
Mar 12, 2011 at 4:02
• @whuber: Thank you for the very basic comment. In the medical field, in emergencies, we are forced to start managing epidemics without thinking of a study. After the epidemic is over, a cursory glance at the data shows us some important conclusion that one treatment given empirically is better than what was given in the other state. This information has to be proven statistically. Then only it will benefit other patients. Please advise me. Mar 12, 2011 at 9:26
• I don't really understand which state is supposed to be superior: rolando2's comment says "State 1", but the original question says "State 2".
– Fr.
Mar 12, 2011 at 18:35
• @DrWho Point taken. But you're definitely not talking about "design," then, are you? Here's another basic point that seems well worth making here: you cannot "prove statistically," after the fact with a sample of convenience, that one treatment was better than the other. The treatment is confounded with the state and all the ways in which the states can differ. All you can do is conclude the data are "suggestive" of a difference.
– whuber
Mar 12, 2011 at 20:25

I'm not sure why time series is not being accepted as a solution to your problem since this is equally spaced chronological data . Confounding variables although unknown in nature can be proxied by both ARIMA structure and/or Detectable Interventions. I'm not sure who is not accepting that time series analysis is appropriate so we disagree with that advice.

Time series methods are not just pure autoregressive in form but easily extend to Polynomial Distributed Lags or ADL in user-specified supporting series such as the number of reported cases.

In my opinion this is an example of a pooled cross-sectional time series problem. Gregory Chow developed a test for the constancy of parameters across groups in 1960 ;

In this case, the test needs to be front-ended with Outlier Detection to ensure a Gaussian set of errors i.e. no proven anomalous data or in other words the error process can't be proven to be Non-Gaussian within each state.

X1 is the number of cases and Y is the number of deaths. X2 is the empirically identified point of anomaly ;(2009 .. period 6 for State1 and 2004 .. period 1 for State2 . Outlier Detection led to identifying one anomalous data point for each state reflecting some unknown background variable thus yielding a more robust estimate of the mortality rates.

## Analysis of State1

State1 Y(T) = -.65649
+[X1(T)][.0046)] CASES +[X2(T)][-1.3608)] PULSE6 I~P00006STATE1 + [A(T)]

Suggesting an unusually low mortality rate for 2009

## Analysis of State2

State2 Y(T) = 123.55
+[X1(T)][(+ .0468)] CASES +[X2(T)][(+ 35.7590)] PULSE1 I~P00001STATE2 + [A(T)]

Suggesting an unusually high mortality rate for 2004

This leads to estimating two cleansed data points

STATE YEAR Y OBSERVED Y ADJUSTED

STATE1 2009 3 4.36

STATE2 2004 254 218.24

Replacing these two observed possibly errant values possibly due to some unspecified concomitant factor (“lurking Variable”) one computes a rate of.0046 for STATE1 and .0468 for STATE2.

The Chow Test for constancy of parameters across groups easily yields a rejection of the null hypothesis of equal coefficients, thus the states can be said to have significantly different mortality rates. Note that even though the “experiment was not a controlled one” it is possible to identify deterministic effects reflecting some uncontrolled input which untreated can distort the analysis.

P.S. I am fully aware that we can’t have 4.36 or 218.24 mortalities in real life !

• @David Reilly: I am amazed at your your commendable and authoritative discussion. One Journal from UK and 1 journal from India refused the paper because I did time series analysis and forecasting to compare what is expected and what was achieved with the new treatment. Mar 15, 2011 at 6:05
• @David Reilly: I wrote this: Selection of the model for forecast generation: Time series expert modeler of IBM SPSS Forecast v19 was used. Both exponential smoothening models and Autoregressive Integrated Moving Average (ARIMA) models were examined. Outliers were detected and prevented from influencing parameter estimates. Maximum number of Lags in ACF and PACF output checked were 24. Stationery R-square (larger values indicate better fit) was the goodness of fit measure for selecting the best fitting forecasting model to generate IRs for 2011 to 2012 for 2 states. Thank you very much. Mar 15, 2011 at 6:06
• @David Reilly: One more question to trouble you a little more. What is the statistical software you have used? I used SPSS Forecast Version 19. I generated a forecast for 2009 & 2010 and showed that the observed death rate is below the Lower Confidence limit in State 1 and so it is significant. Mar 15, 2011 at 6:19
• @DrWho: I am a developer of time series software. The product name is AUTOBOX and is available from autobox.com. I hope that this comment is not perceived as being "too commercial" as that is not my intent at all. Please feel free to contact me via that site. Mar 15, 2011 at 10:55
• @IrishStat: Thank you very much for the very useful reply. Your answer was very useful. I am still reading it to understand it completely because I am a beginner in Statistics with negligible knowledge of Mathematics. I am sorry I missed your reply for so long. I read about you in your website. I must congratulate you for being so active and prompt at 72. Mar 16, 2011 at 13:54

I might have wrongly understood the data (see my comment above), but what would be so wrong in simply t-testing the difference in means of the deaths-to-cases ratios between the two series, assuming that the states are clearly independent from each other?

A t-test shows that State 2 performs far better than State 1 in preventing deaths in its cases of the disease:

year    state   ratio
2004    2   7.968504
2004    1   225
2005    2   9.464115
2005    1   242.6
2006    1   250.75
2006    2   10.57143
2007    1   237.5
2007    2   9.285141
2008    2   7.756345
2008    1   293
2009    1   364
2009    2   8.25
2010    2   9.284404
2010    1   329


Given the huge difference between the two series, and given that you cannot design a proper experiment, it seems to me that no superior level of "proof" is really needed to proclaim one state's approach superior to the other.

• You're right; sometimes the results are so obvious that applying a formal test would be overkill--and actually woudl look suspicious ("what's this guy trying to hide with his statistics?"). But by themselves, knowing only what we have been told so far, these data are not convincing. For instance, maybe the treatment was completely without effect but there happened to be two strains of this disease in State 2 (but not in State 1) and the second strain was much more virulent. The point is that any "proof" must rest on scientific argument, not a statistical one.
– whuber
Mar 12, 2011 at 22:42
• Did you mean that "State 1 performs better"? Mar 13, 2011 at 3:19
• @SheldonCooper: Thank you very much for pointing out the mistake. I have rectified my mistake. Mar 13, 2011 at 4:17
• @whuber: The two states are neighboring states. People can walk in to the other state freely. The cases are investigated epidemiologically and virologically and found to be similar. Thank you very much for the excellent comments. They are of great helpto me. Mar 13, 2011 at 4:20
• @whuber: I heartily agree with your argument. I just read that quote by Ronald Fisher in another discussion: "To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of".
– Fr.
Mar 13, 2011 at 15:33

The final anlalysis was published in http://childscience.org/html/jpn/v10n4_3.html . This is an example of the power of StackExchange to bring together a keen researcher and a subject matter expert. Title of paper "Reducing case fatality rate of acute encephalitis syndrome in developing countries" . Journal of Pediatric Neurology

• Thank you very much StackExchange and IrishStat. I can never forget the guidance and help from you. This paper will help the Governments of developing countries in making health policy decisions and save millions of children. Dec 20, 2012 at 14:25