# Validation of data partitioning

I've some numerical sequence of data monitored over time $v(i)$, which belongs to one of two classes, 0 or 1. I monitor the variable several times, obtaining a sequence associated to 0 or 1. I use part of the cases in which $v(i)$ belongs to class 0 to have a profile/baseline (training data/cases). I then use the cases in which $v(i)$ belongs to 1 to see if there is some difference from the baseline (test data/cases). I end then in computing a metric, based on the information showed by $v(i)$, namely if it showed some "difference" when the case was a 1, or a 0 (error), and the viceversa considering the class 0.

So from the whole cases $N$ I get just one value $M(v(i),N)$. But this value is just a statistic.

My question is: How to guess the distribution of this metric? I thought to partition the "test data/cases" in $k$ groups (keeping the proportions), and then estimate the distribution of $M(v(i),N/k)$ through a $T$-test, or something similar. But does it make really sense? Is it a valid way to infer the distribution of $M$?

Thanks in advance for the tips and discussion I will (in case) receive :)

-
Are you actually interested in the distribution of the metric or in whether there is a significant difference between the distribution of the metric when the class is 0 and when it's 1? There's a subtle difference between the two; you can test for a significant difference even if you don't know the underlying distribution. –  jbowman Dec 3 '11 at 17:25