# How to tell if there is any improvement, statistically

My statistic knowledge only goes so far as to understand those basic statistic terms, instead of applying them, yet this is a questions on applying statistic knowledge into real world case.

For statistic terms like Mean, Median, & Mode etc, my knowledge only limits to a simple distribution like this:

However, from histogram, I found the actual distribution that I'm getting is more or less like this:

It's not 100% like this, but the point is that there are multiple distributions apparent within the histogram. I.e., I'm able to identify observations from several different populations from the histograms.

UPDATE:

To give a more detailed explanation, think of the time when manufacturing was label intensive -- each person carries out the same task at different speeds than others; Moreover, each time your measure their speed to finish the work, the measures are slightly difference than others even for the same person. This is the situation. The goal is to measure if a new approach can improve the overall speed or not.

UPDATE 3:, Having provided "UPDATE 2" as per request, I'd like to stress that my focus is not on any specific set of data, or any specific scenario, as you can see that I've kept changing the situations that my question would apply. The reason is that, I want to know, conceptually, how to solve all these kind of problems statistically. I.e., I believe the scenario I describe above is a rather common one. I'd be surprise that there is no existing statistical theory/practice that covers this -- finding the "truth" while eliminating the impact of the outliers. Intuitively, I was first thinking, if the variation is so big, and representing them with a single line (value of Mean or Median etc) is not enough, then how about representing them with a band? I.e., there ought to be some statistical theory/practice that covers this common problem. Now I'm forced to pick up statistics books again myself, and I've started reading for several days. For the representing them with a band idea, I found the interquartile range somewhat close to what I was thinking. It is resistant to the influence of outliers, but I'm more interested in value of interquartile's Q1 and Q3, than their differences, i.e., the band rather than the gap. Further on finding exiting statistical theory/practice, I also found Median Absolute Deviation (MAD), which is a robust measure of central tendency. I also found the term Robust statistics which are "statistics with good performance for data drawn from a wide range of non-normally distributed probability distributions" (ref: here), which looks to me quite close to what I'm searching for. These are the finding I got from these several days of quick brush up, meaning, I am now knowing more terms now, but still not know how applicable they are to the above problem. These, are what I'm asking for, the exiting statistical theory/practice that deal with this situation. Please help. Talking about different situations that my question would apply, now here is another one -- think of the case that different people would bid on different prices for a stock. This is for sure a non-normally distributed probability distributions. So in scenario, my question is, when situation changed, are most people bidding more, or bidding less? and how much differences (in %) than before. This the exact same question I'm asking here.

UPDATE 2:, actual data

The three sets at the end of this post are the actual data I'm talking about.

• The sample numbers are only around 50, too low for the multiple distributions case, but if you plot the histogram, you can see the hint that there are multiple distributions in them.
• The Val is my target. There are many many factors contributing to its variation. I think the major one is cause by the random sampling method I explained below, and I believe it is a running average of all players in the game, just like averaging all bidders bidding price in the stock. Some bidders bid aggressively (higher value) and some don't. But at the end of the day, its measured value is all that I collect and compare.
• All three sets are taken under the exactly same condition. Measuring using Mean, or even Medium, the variation alone is too much to be useful to gauge the improvements between different conditions.

Given a situation like this, how can I tell if there is any improvement between different tests?

The challenges here are,

• We are only picking a limited number of samples, as the total samples are too big
• To try to accurately represent the whole spectrum, we use random sampling
• This random sampling alone will cause problem for us even if everything else remaining the same. Sometime we see changes between each sampling are over 15% (in Mean, yeah I know, we shouldn't use Mean)
• Thus so far I just cannot tell for sure if things get any better after an improvement attempt has been made, because an improvement rate of 10~15% is already a significant improvement. However, because of the variation above, I'm really not sure, which is contributing to the change.

So, the question is, for situation like this,

• What the systematic method(s) we should use to gauge the improvements.
• If Mean is the worst statistic measurement to use, then what measurements are best suit for cases like this?

Measure 1

Elapsed,Val
00:00:00,31.323
00:00:08,17.802
00:00:09,60.685
00:00:13,9.39
00:00:38,12.458
00:00:39,7.036
00:00:49,6.547
00:00:49,14.152
00:01:12,7.713
00:01:19,6.292
00:01:26,6.746
00:01:31,5.998
00:01:48,14.821
00:01:57,22.89
00:02:04,9.221
00:02:22,25.383
00:02:30,7.117
00:02:39,15.308
00:02:48,11.752
00:03:08,13.713
00:03:12,35.665
00:03:17,10.309
00:03:38,36.091
00:03:39,7.95
00:03:40,17.545
00:04:06,10.355
00:04:12,31.293
00:04:26,6.031
00:04:28,21.111
00:04:31,9.748
00:04:46,23.068
00:05:01,12.469
00:05:04,21.264
00:05:12,8.657
00:05:22,14.197
00:05:44,8.728
00:05:44,8.73
00:05:56,13.121
00:06:15,14.343
00:06:20,6.943
00:06:32,23.77
00:06:37,7.579
00:06:49,6.512
00:06:56,7.273
00:07:08,6.194
00:07:22,5.039
00:07:25,6.339
00:07:41,19.308
00:07:46,15.539
00:07:52,8.482
00:08:10,14.027
00:08:19,10.463
00:08:26,14.658
00:08:34,4.329
00:08:51,10.613


Measure 2

Elapsed,Val
00:00:00,20.819
00:00:03,16.926
00:00:08,11.343
00:00:17,6.975
00:00:29,13.553
00:00:35,7.082
00:00:49,8.147
00:01:03,12.317
00:01:07,6.171
00:01:15,6.291
00:01:33,10.982
00:01:39,6.609
00:01:54,7.276
00:01:56,13.104
00:02:05,13.122
00:02:07,6.455
00:02:23,11.779
00:02:41,8.026
00:02:53,7.798
00:02:55,15.992
00:03:02,6.863
00:03:13,12.208
00:03:25,7.654
00:03:34,7.559
00:03:44,13.802
00:03:54,11.604
00:04:03,6.395
00:04:15,6.894
00:04:30,9.393
00:04:30,14.638
00:04:55,7.87
00:04:59,10.674
00:05:01,10.533
00:05:18,5.466
00:05:25,5.351
00:05:37,7.138
00:05:40,4.688
00:05:54,7.194
00:06:10,11.131
00:06:20,6.128
00:06:28,16.418
00:06:43,18.778
00:06:53,20.298
00:07:01,4.311
00:07:15,14.836
00:07:24,12.203
00:07:28,13.222
00:07:32,8.896
00:07:44,10.438
00:08:04,26.271
00:08:12,18.06
00:08:20,9.327
00:08:32,7.522
00:08:36,6.431
00:08:52,13.165


Measure 3

Elapsed,Val
00:00:00,9.778
00:00:11,6.743
00:00:13,8.393
00:00:32,8.085
00:00:37,5.142
00:00:54,4.717
00:01:07,5.378
00:01:16,13.073
00:01:24,6.612
00:01:28,7.67
00:01:34,5.738
00:01:48,23.959
00:01:52,5.774
00:02:02,5.031
00:02:11,4.419
00:02:26,6.532
00:02:41,3.9
00:02:43,13.194
00:03:00,6.707
00:03:07,6.283
00:03:20,8.528
00:03:29,22.631
00:03:37,6.113
00:03:55,16.971
00:04:04,8.699
00:04:21,6.023
00:04:29,12.985
00:04:30,3.607
00:04:42,49.959
00:04:45,6.279
00:04:54,5.978
00:05:03,5.296
00:05:18,11.584
00:05:31,9.281
00:05:37,4.632
00:05:51,5.445
00:05:54,18.232
00:06:08,6.94
00:06:21,5.182
00:06:26,23.377
00:06:32,13.823
00:06:45,7.367
00:07:01,8.115
00:07:05,8.361
00:07:19,13.313
00:07:24,7.255
00:07:44,14.877
00:07:49,16.052
00:07:57,5.811
00:08:12,15.962
00:08:20,7.89
00:08:32,4.374
00:08:41,7.653
00:08:43,6.85
00:09:02,5.085


Thanks

• Welcome to Cross Validated! The histograms suggest you're able to identify observations from several different populations - is that in fact the case? Exactly what's going on with the sampling scheme is too vaguely described - you'll need to add more explanation, or perhaps return to it in a subsequent question. Commented Jan 12, 2017 at 16:54
• Yes, OP amended reflecting that. Thanks @Scortchi.
– xpt
Commented Jan 12, 2017 at 16:57
• So why not consider each population separately? Commented Jan 12, 2017 at 16:58
• See the added "to give more explanation" section. It's the same group of people -- just statistically they can be grouped, not individually, because there are many factor contributing to such variation, not just one or two. So the only thing I can do is to view them statistically. @Scortchi
– xpt
Commented Jan 12, 2017 at 17:12
• It's tempting to see this as simply comparing the distributions from independent observations of Val under three conditions: Measure 1, Measure 2, & Measure 3. But your comments about measurements of the same person & different people, & about running averages of players, & that the observations appear to be time series, hint at more structure than you're letting on. That aside, the distributions do seem to contain outliers & be of rather different shapes: you might want to consider what would constitute "improvement" in the population distributions if you knew them for sure. Commented Jan 12, 2017 at 23:29

## 1 Answer

Let me try to restate your problem using the most generic language possible, so as to direct you to appropriate ideas and resources.

You have a longitudinal study where there is substantial heterogeneity across the units (which I take to be people) in your population. You apply an experimental intervention (aka, treatment) to the units, and observe effects upon the time series data that each unit generates. The effects of these interventions are superimposed upon a trend in each time series, so that you encounter a problem of detrending the data to identify separately the treatment effect.

Furthermore, you have also sampled randomly from your design for the common reason that it is costly (e.g., in terms of time and effort) to collect all of the potentially available data.

Given the complexity of your problem, I would advise that it is premature to leap to an application of statistical methods. Your first step should be graphical data exploration. There are many tools to choose from; this question includes a discussion with many links. An excellent place to start is with the sort of plot you see here. Once you can post plots like those, CV will be in a much better position to suggest appropriate statistical models and techniques to answer the scientific questions you have. I'm certain that efforts toward data visualization will help you to focus your questions, and articulate them more pointedly.

• Two thumbs up for this approach/answer, as I'm practically blind as what I should be looking for/at. Only one clarification: "Once you can post plots like those, CV will be ...", what does CV mean here?
– xpt
Commented Sep 10, 2017 at 18:40
• Oh, got it myself, CV -- Cross Validated. I was thinking it has something to do with the site, but the site is called "stats.stackexchange.com"
– xpt
Commented Sep 10, 2017 at 18:43
• Actually, you're quite right. CV could also mean coefficient of variation; I will be more careful about that acronym in future! Commented Sep 10, 2017 at 19:20