# Effectiveness test of intervention on number of cases

Apologies if I'm phrasing this incorrectly,

I want to model the effectiveness of an intervention measure on the number of patients diagnosed with a virus (COVID-19). A hypothesis test of sorts comes to mind, but I'm not sure how to account for the exponential growth of the virus in the time variable

More clearly, Say I have n days pre-intervention,and n days after (and n=10):

                                                         <INTERVENTION DAY 10 ONWARDS>
day_num            0   1   2   3    4    5    6     7     8     9     10    11     12     13     14     15     16     17     18     19
total_cases_a_day  10  20  40  80  160  320  640  1280  2560  5120  8225  11178  13987  16659  19201  21619  23919  26107  28188  30168



Data above is mocked with doubling prior to day 10, and exponential decay on the increase of cases per day after that. Code below for perusal.

How do i quantify the impact of the intervention? eg, a hypothesis test or something to show that it worked? Would a regression help? eg,

log(num cases) ~ intervention_measure + time. Where intervention is binary? or continuous depending on strength of intervention..?

Thank you!!

Code to create mock data


import pandas as pd
import numpy as np
pd.options.display.max_columns = 30

days = list(range(20))

c = 2
before_cases = [10*c**i for i in range(10)]

def exponential_decay(starting_val,lambda_val,t):
return int(starting_val * (np.e)**(-(lambda_val)*t))

lambda_val = 0.05
starting_val = before_cases[-1]

after_increment_a_day = [exponential_decay(starting_val,
lambda_val,
i) for i in range(10,20)]

after_cases = []
current = starting_val
for entry in after_increment_a_day:
after_cases.append(current + entry)
current += entry

total_cases = before_cases + after_cases
df = pd.DataFrame({"total_cases_a_day":total_cases,
"day_num":list(range(len(total_cases)))})

df = df.set_index('day_num')
df = df.T
df
$$$$


I assume log(num cases) ~ intervention_measure + time` includes an intercept (this is the default in R, do not know about Python). If you were willing to entertain an exponential trend, that would seem right to me: but this is clearly untenable beyond a certain number of days in a finite population.

You write about exponential decay after day 10. Induced by the intervention, or independent of the intervention? In the later case (which might be the case in the population develops some natural inmunity as more people are infected and fight the infection) you should include that decay into your baseline model, so that the intervention only picks up any addittional slowdown not attributable to natural causes.

• Im open to anything, it was an open question with one example. The exponential decay is just to mock the cases having a decreasing trend. It doesnt have to be so in nature. I just want to find a way to quantify this?
– Wboy
Apr 10, 2020 at 16:59

A problem with statistical hypothesis tests is that they are directly testing the models and only indirectly your interpretations of these models.

Say you determine the absence of some effect is not present in your model, then either the effect is not present or the effect is present but you modelled it wrongly.

It can go both ways:

• In the case of interventions you need to consider a lag between the time the interventions started and the time series starts to show the difference. If you do not do this, then you may wrongly conclude absence of the effect.
• Also most interventions are not part of a controlled study and multiple factors that can have an influence change at the same time. And you may observe a change in the time series that you wrongly ascribe to the intervention.
• Hello, it's fine, this isnt a clinical study. It's more about finding whether lockdowns are effective in reducing the number of infections. Could you advise how to do this?
– Wboy
Apr 16, 2020 at 6:51
• @Wboy The data like cases per day are not useful to find out the effect of a lock down: 1) There are too many variables that change at the same time. 2) You will only observe correlation, causation remains undetermined. 3) The sampling of cases is very biased. To solve this you would have to do a controlled test. For instance, you could ask the presidents of Brazil or the USA (I imagine they can be convinced to follow this plan) to select randomly villages that they divide into open and locked down. After that, infect them and observe and compare the growth numbers for the different groups. Apr 16, 2020 at 7:45
• @Wboy a more practical example is the fitting of SIR compartment models to the case numbers. This will give a reproduction number $R_0 \approx 1$ but it is a meaningless result. The SIR model doesn't take into account the other many influences. What the model tells you, doesn't need to be realistic. Whenever you have a significant result it means that either the effect is true or the effect is not true but the model is wrong. And similarly whenever you find an insignificant result it means that either the effect is not present or the effect is present but the model is wrong/insufficient. Apr 16, 2020 at 7:56
• The comment about the presidents was a sarcastic joke. But controlled studies are the way to find out more knowledge about the spread of virusses. Obviously you can not perform a test simulating an entire lock down situation. That would be unethical. However one can perform laboratory tests that study the nature of the virus (e.g. how long it remains active depending on circumstances, and how it effectively spreads). From those results one can (theory based) deduce what the effects of a lock down could be. Apr 16, 2020 at 8:04
• I know the ideal case would be to A/B test it with different villages, but thats not really the point here. Neither is a perfect study. In my case above, assuming that the only active factor is whether there being a lockdown or not, how can I quantify whether the lockdown worked?
– Wboy
Apr 16, 2020 at 8:08