I've some numerical sequence of data monitored over time $v(i)$, which belongs to one of two classes, 0 or 1. I monitor the variable several times, obtaining a sequence associated to 0 or 1. I use part of the cases in which $v(i)$ belongs to class 0 to have a profile/baseline (training data/cases). I then use the cases in which $v(i)$ belongs to 1 to see if there is some difference from the baseline (test data/cases). I end then in computing a metric, based on the information showed by $v(i)$, namely if it showed some "difference" when the case was a 1, or a 0 (error), and the viceversa considering the class 0.
So from the whole cases $N$ I get just one value $M(v(i),N)$. But this value is just a statistic.
My question is: How to guess the distribution of this metric? I thought to partition the "test data/cases" in $k$ groups (keeping the proportions), and then estimate the distribution of $M(v(i),N/k)$ through a $T$-test, or something similar. But does it make really sense? Is it a valid way to infer the distribution of $M$?
Thanks in advance for the tips and discussion I will (in case) receive :)