What metric should I use to determine a significant effect?

Question

I am not a statistician and hope someone can point me towards the right direction. I have some time series data grouped into three classes like this:

Time Period 1       Time Period 2     Time Period 3
-------------------------------------------------
[1,2,3,4,5,6...]    [12,13,14,15]     [17,3,1,3,4...]
[1,3,5,6,8,9...]    [6,8,7,9,6,4]     [1,2,5,7,3,2...]
[9,8,9,9,8,9...]    [3,1,1,2,1,2]     [7,8,9,9,9,8...]

The dots indicate that I have significantly more values for Time Period 1 and Time Period 3 than for Time Period 2. I am trying to define events of "interest". Interest is when there is a significant change in the transitions from Time Period 1 to Time Period 2 and Time Period 2 to Time Period 3. Of course, there could be a knob for determining what I mean for significant.

What I am looking for is a good metric that tells me if an event is of potential interest. Obviously, average would not do good due to outliers so all I could think of was the median which seemed like a good one i.e. if there is a significant change in the median from Time Period 1 to Time Period 2, then this transition is of interest.

While this metric is working out for me, I am curious if there is a more structured/formal approach to deriving a metric that is better than the median. As of now, the problem formulation is open as well so any suggestions/constructive criticisms are greatly appreciated.

Answer 1

An idea is to try a set of different metrics, running them through the same data set and then compare their ratings (places of "interesting things") with those of human classifiers. Then you can select the metric that is the closest to the rating of humans either based on correctly identified instances (true positives) or based on the number of incorrectly missed instanced (false negatives).

Of course, this is basically just guess work. You could also try to approach the task with theory, find out what the numbers mean and what exactly an event of interest is. If you have a good specification of these questions you basically have your classifier.