Statistical test to compare multiple treatments on the same 'individual' over time

Question

I'm looking for a statistical test or tests to compare the effects of different 'treatments' on a single 'individual' over time. The single 'individual' could be one person or a group acting as a single entity (e.g. a team, a company, etc.). A simple example is comparing the effects of two different exercise regimes on one person's weight loss.

The main concern I have with using a 'standard' statistical test (in this case, something like a t-test, Mann-Whitney U test or Kolmogorov-Smirnov Test seems appropriate) is the obvious non-independence of the data.

I could check for independence by running some form of ABAB study (i.e. alternating the regimes several times to check that they are not interacting). This seems to be the approach that the author of this 'Self Experimentation' paper (Roberts, S.) seems to take. He then uses repeated chi-squared tests (first A against first B, second A against first B, etc.). Is it a valid statistical approach to perform multiple tests on the same data or does this increase the chance of type I errors as indicated in this paper?

The more often one analyses the accumulating data, the greater the chance of eventually and wrongly detecting a difference, thereby drawing incorrect conclusions from the trial.

Should I worry about the non-independence? Can I simply perform multiple tests against the same data? Is there a specialised test for this kind of experiment?

Answer 1

If the time aspect is very important then you may want to consider some type of structured dependency model that can deal with a time series, but if I'm understanding your question correctly here's my answer.

The problem with conducting those standard statistical tests repeatedly is called alpha inflation or the problem of multiple comparisons.

Analysis of Variance is a procedure that allows conducting those standard tests simultaneously. The most basic version is a generalized t-test. There are variations of ANOVA like Kruskal-Wallis which is nonparametric. ANOVA is discussed thoroughly on this site. Here is an example of how to do it in SAS, SPSS, and R.

Since you're measuring the same individual repeatedly you'll want to use a repeated measures design. If you're testing multiple hypotheses you'll want to also use a correction factor like Bonferroni, but as far as your question seems to be concerned this may not be the case so this last part may be unnecessary. rANOVA can be sensitive to assumption violations though so if you see your data violates them you may need to consider some other linear model.

I hope this helps. More information about your problem would be useful to give more specific of an answer. Feel free to follow up if this leads to other questions.

Answer 2

It is good to assume that the time series data would not be independent. It is highly likely hat there would be some serial correlation (auto-correlation) within the subsequent measurements of each treatment. You may estimate time series models for each treatment and then compare the time series to see if the treatments differ in their efficacy.

Here are the resources for comparing time series that I wrote as answer to another question.

Answer 3

If you are using survival analysis, you could introduce a time dependent covariate to model the problem. This method can accept time dependent variants - in your case, you could say that patient A was in treatment X from day 1 to day 60 and was in treatment Y from day 61 to day 100.

Here are some useful links. https://cran.r-project.org/web/packages/survival/vignettes/timedep.pdf https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6015946/