I'm looking for a statistical test or tests to compare the effects of different 'treatments' on a single 'individual' over time. The single 'individual' could be one person or a group acting as a single entity (e.g. a team, a company, etc.). A simple example is comparing the effects of two different exercise regimes on one person's weight loss.
The main concern I have with using a 'standard' statistical test (in this case, something like a t-test, Mann-Whitney U test or Kolmogorov-Smirnov Test seems appropriate) is the obvious non-independence of the data.
I could check for independence by running some form of ABAB study (i.e. alternating the regimes several times to check that they are not interacting). This seems to be the approach that the author of this 'Self Experimentation' paper (Roberts, S.) seems to take. He then uses repeated chi-squared tests (first A against first B, second A against first B, etc.). Is it a valid statistical approach to perform multiple tests on the same data or does this increase the chance of type I errors as indicated in this paper?
The more often one analyses the accumulating data, the greater the chance of eventually and wrongly detecting a difference, thereby drawing incorrect conclusions from the trial.
Should I worry about the non-independence? Can I simply perform multiple tests against the same data? Is there a specialised test for this kind of experiment?