I want to test the effect of a treatment on a continuous outcome (specifically interested in increase in average in the treatment group). The experiment is run over a period of time. But the problem here is that the relative values of the outcome (and hence the daily average) tend to vary over time (and is mostly random variation day over day).

What I am interested is, finding out if there is a significant increase in the mean of the outcome due to the treatment. The sample size for the treatment is relatively large enough for a single day, but are not constant over days (ratio of treatment size to control size is constant however).

What is the best way to test this, given the random time effect on both treatment and control ? Can the percentage change relative to the control, be used a metric in some appropriate test, given that the treatment sample size is not constant over days ?

EDIT: Further clarification of the situation

It'is better not to think of this as an individual. Let's say there is an environment and lot of variables that operate within it. The environment has an outcome. (Think of it as production volume, revenue, sales, visits etc,..). Now we are trying a treatment on the environment by affecting one variable and want to see if that variable has a positive effect on the outcome. We are measuring the outcome on a daily basis and want to test if the mean value of the outcome is increased in the treatment group. But the problem is that, even in a situation where you don't have a treatment, the net outcome and the mean outcome fluctuate on a daily basis by random factors. So the issue at hand is to devise a proper test to study the treatment, while somehow accounting for this random variability in the outcome. Would appreciate if there are any suggestions for this case.

  • $\begingroup$ Welcome to Stats.SE. Can you please explain a bit more what you have. If I understood correctly, each individual $j$ belongs either to the treatment group or to the control group. Furthermore, you have a continuous outcome for each individual that is measured along the time, so you have $x_j(t)$. What I did not understand is what you want to do to the measures $x_j(t)$. Do you want to check if there were some change in those measures at the end of the treatment, or at a fixed time after the beginning, or are you thinking about something else? $\endgroup$ Apr 28 '19 at 0:25
  • $\begingroup$ @Ertxiem, edited for further clarification $\endgroup$
    – sibiyes
    May 2 '19 at 1:51

It seems to me that a regression model might be adequate in your case.

In the regression model, the outcome $y$ is the dependent variable and you have several independent variables $x_j$, the treatment variable $x_1$ and the other variables that may act as confounders (variables that may also affect $y$ besides $x_1$). The choice of model depends on the characteristics of your data, I'll present two examples.

One of the simplest regression models is the linear regression, where the outcome $y$ is modelled as a linear combination of the independent variables $x_j$. If the effect of the treatment $x_1$ is statistically significant, you could infer that treatment affects the outcome.

However, it seems to me that the outcome $y(t)$ is a time series and there might be a memory effect, i.e., $y(t+1)$ is related with $y(t)$. In this case, it might be more appropriate to look into more complex models, like the autoregressive integrated moving average - ARIMA model.

  • $\begingroup$ I did think that way, but wanted to explore a possibility of using an appropriate testing method with a statistic generated from the sample. Would it make sense to compute percent change for the test group relative to the control (u_test - u_control/u_control) and test if that is significantly greater than 0. One of the issues is that the sample size for the control is much greater than the test (since control is the real world we operate in) and the test sample size varies day to day. (data is collected by sampling a percentage of overall for test group). $\endgroup$
    – sibiyes
    May 2 '19 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.