Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm looking for a statistical test or tests to compare the effects of different 'treatments' on a single 'individual' over time. The single 'individual' could be one person or a group acting as a single entity (e.g. a team, a company, etc.). A simple example is comparing the effects of two different exercise regimes on one person's weight loss.

The main concern I have with using a 'standard' statistical test (in this case, something like a t-test, Mann-Whitney U test or Kolmogorov-Smirnov Test seems appropriate) is the obvious non-independence of the data.

I could check for independence by running some form of ABAB study (i.e. alternating the regimes several times to check that they are not interacting). This seems to be the approach that the author of this 'Self Experimentation' paper (Roberts, S.) seems to take. He then uses repeated chi-squared tests (first A against first B, second A against first B, etc.). Is it a valid statistical approach to perform multiple tests on the same data or does this increase the chance of type I errors as indicated in this paper?

The more often one analyses the accumulating data, the greater the chance of eventually and wrongly detecting a difference, thereby drawing incorrect conclusions from the trial.

Should I worry about the non-independence? Can I simply perform multiple tests against the same data? Is there a specialised test for this kind of experiment?

share|improve this question
It would help to know more about your specific situation and the specific question you want to answer. If you do an ABBA study (perhaps repeatedly) you might be able to calculate the difference between the first A and B, the difference between the second A and B, and so on. You could then do a t test to see if these differ from zero. – Joel W. Aug 23 '12 at 12:35
up vote 2 down vote accepted

If the time aspect is very important then you may want to consider some type of structured dependency model that can deal with a time series, but if I'm understanding your question correctly here's my answer.

The problem with conducting those standard statistical tests repeatedly is called alpha inflation or the problem of multiple comparisons.

Analysis of Variance is a procedure that allows conducting those standard tests simultaneously. The most basic version is a generalized t-test. There are variations of ANOVA like Kruskal-Wallis which is nonparametric. ANOVA is discussed thoroughly on this site. Here is an example of how to do it in SAS, SPSS, and R.

Since you're measuring the same individual repeatedly you'll want to use a repeated measures design. If you're testing multiple hypotheses you'll want to also use a correction factor like Bonferroni, but as far as your question seems to be concerned this may not be the case so this last part may be unnecessary. rANOVA can be sensitive to assumption violations though so if you see your data violates them you may need to consider some other linear model.

I hope this helps. More information about your problem would be useful to give more specific of an answer. Feel free to follow up if this leads to other questions.

share|improve this answer

It is good to assume that the time series data would not be independent. It is highly likely hat there would be some serial correlation (auto-correlation) within the subsequent measurements of each treatment. You may estimate time series models for each treatment and then compare the time series to see if the treatments differ in their efficacy.

Here are the resources for comparing time series that I wrote as answer to another question.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.