My statistic knowledge only goes so far as to understand those basic statistic terms, instead of applying them, yet this is a questions on applying statistic knowledge into real world case.
However, from histogram, I found the actual distribution that I'm getting is more or less like this:
It's not 100% like this, but the point is that there are multiple distributions apparent within the histogram. I.e., I'm able to identify observations from several different populations from the histograms.
To give a more detailed explanation, think of the time when manufacturing was label intensive -- each person carries out the same task at different speeds than others; Moreover, each time your measure their speed to finish the work, the measures are slightly difference than others even for the same person. This is the situation. The goal is to measure if a new approach can improve the overall speed or not.
UPDATE 3:, Having provided "UPDATE 2" as per request, I'd like to stress that my focus is not on any specific set of data, or any specific scenario, as you can see that I've kept changing the situations that my question would apply. The reason is that, I want to know, conceptually, how to solve all these kind of problems statistically. I.e., I believe the scenario I describe above is a rather common one. I'd be surprise that there is no existing statistical theory/practice that covers this -- finding the "truth" while eliminating the impact of the outliers. Intuitively, I was first thinking, if the variation is so big, and representing them with a single line (value of Mean or Median etc) is not enough, then how about representing them with a band? I.e., there ought to be some statistical theory/practice that covers this common problem. Now I'm forced to pick up statistics books again myself, and I've started reading for several days. For the representing them with a band idea, I found the interquartile range somewhat close to what I was thinking. It is resistant to the influence of outliers, but I'm more interested in value of interquartile's Q1 and Q3, than their differences, i.e., the band rather than the gap. Further on finding exiting statistical theory/practice, I also found Median Absolute Deviation (MAD), which is a robust measure of central tendency. I also found the term Robust statistics which are "statistics with good performance for data drawn from a wide range of non-normally distributed probability distributions" (ref: here), which looks to me quite close to what I'm searching for. These are the finding I got from these several days of quick brush up, meaning, I am now knowing more terms now, but still not know how applicable they are to the above problem. These, are what I'm asking for, the exiting statistical theory/practice that deal with this situation. Please help. Talking about different situations that my question would apply, now here is another one -- think of the case that different people would bid on different prices for a stock. This is for sure a non-normally distributed probability distributions. So in scenario, my question is, when situation changed, are most people bidding more, or bidding less? and how much differences (in %) than before. This the exact same question I'm asking here.
UPDATE 2:, actual data
The three sets at the end of this post are the actual data I'm talking about.
- The sample numbers are only around 50, too low for the multiple distributions case, but if you plot the histogram, you can see the hint that there are multiple distributions in them.
Valis my target. There are many many factors contributing to its variation. I think the major one is cause by the random sampling method I explained below, and I believe it is a running average of all players in the game, just like averaging all bidders bidding price in the stock. Some bidders bid aggressively (higher value) and some don't. But at the end of the day, its measured value is all that I collect and compare.
- All three sets are taken under the exactly same condition. Measuring using Mean, or even Medium, the variation alone is too much to be useful to gauge the improvements between different conditions.
Given a situation like this, how can I tell if there is any improvement between different tests?
The challenges here are,
- We are only picking a limited number of samples, as the total samples are too big
- To try to accurately represent the whole spectrum, we use random sampling
- This random sampling alone will cause problem for us even if everything else remaining the same. Sometime we see changes between each sampling are over 15% (in Mean, yeah I know, we shouldn't use Mean)
- Thus so far I just cannot tell for sure if things get any better after an improvement attempt has been made, because an improvement rate of 10~15% is already a significant improvement. However, because of the variation above, I'm really not sure, which is contributing to the change.
So, the question is, for situation like this,
- What the systematic method(s) we should use to gauge the improvements.
- If Mean is the worst statistic measurement to use, then what measurements are best suit for cases like this?
Elapsed,Val 00:00:00,31.323 00:00:08,17.802 00:00:09,60.685 00:00:13,9.39 00:00:38,12.458 00:00:39,7.036 00:00:49,6.547 00:00:49,14.152 00:01:12,7.713 00:01:19,6.292 00:01:26,6.746 00:01:31,5.998 00:01:48,14.821 00:01:57,22.89 00:02:04,9.221 00:02:22,25.383 00:02:30,7.117 00:02:39,15.308 00:02:48,11.752 00:03:08,13.713 00:03:12,35.665 00:03:17,10.309 00:03:38,36.091 00:03:39,7.95 00:03:40,17.545 00:04:06,10.355 00:04:12,31.293 00:04:26,6.031 00:04:28,21.111 00:04:31,9.748 00:04:46,23.068 00:05:01,12.469 00:05:04,21.264 00:05:12,8.657 00:05:22,14.197 00:05:44,8.728 00:05:44,8.73 00:05:56,13.121 00:06:15,14.343 00:06:20,6.943 00:06:32,23.77 00:06:37,7.579 00:06:49,6.512 00:06:56,7.273 00:07:08,6.194 00:07:22,5.039 00:07:25,6.339 00:07:41,19.308 00:07:46,15.539 00:07:52,8.482 00:08:10,14.027 00:08:19,10.463 00:08:26,14.658 00:08:34,4.329 00:08:51,10.613
Elapsed,Val 00:00:00,20.819 00:00:03,16.926 00:00:08,11.343 00:00:17,6.975 00:00:29,13.553 00:00:35,7.082 00:00:49,8.147 00:01:03,12.317 00:01:07,6.171 00:01:15,6.291 00:01:33,10.982 00:01:39,6.609 00:01:54,7.276 00:01:56,13.104 00:02:05,13.122 00:02:07,6.455 00:02:23,11.779 00:02:41,8.026 00:02:53,7.798 00:02:55,15.992 00:03:02,6.863 00:03:13,12.208 00:03:25,7.654 00:03:34,7.559 00:03:44,13.802 00:03:54,11.604 00:04:03,6.395 00:04:15,6.894 00:04:30,9.393 00:04:30,14.638 00:04:55,7.87 00:04:59,10.674 00:05:01,10.533 00:05:18,5.466 00:05:25,5.351 00:05:37,7.138 00:05:40,4.688 00:05:54,7.194 00:06:10,11.131 00:06:20,6.128 00:06:28,16.418 00:06:43,18.778 00:06:53,20.298 00:07:01,4.311 00:07:15,14.836 00:07:24,12.203 00:07:28,13.222 00:07:32,8.896 00:07:44,10.438 00:08:04,26.271 00:08:12,18.06 00:08:20,9.327 00:08:32,7.522 00:08:36,6.431 00:08:52,13.165
Elapsed,Val 00:00:00,9.778 00:00:11,6.743 00:00:13,8.393 00:00:32,8.085 00:00:37,5.142 00:00:54,4.717 00:01:07,5.378 00:01:16,13.073 00:01:24,6.612 00:01:28,7.67 00:01:34,5.738 00:01:48,23.959 00:01:52,5.774 00:02:02,5.031 00:02:11,4.419 00:02:26,6.532 00:02:41,3.9 00:02:43,13.194 00:03:00,6.707 00:03:07,6.283 00:03:20,8.528 00:03:29,22.631 00:03:37,6.113 00:03:55,16.971 00:04:04,8.699 00:04:21,6.023 00:04:29,12.985 00:04:30,3.607 00:04:42,49.959 00:04:45,6.279 00:04:54,5.978 00:05:03,5.296 00:05:18,11.584 00:05:31,9.281 00:05:37,4.632 00:05:51,5.445 00:05:54,18.232 00:06:08,6.94 00:06:21,5.182 00:06:26,23.377 00:06:32,13.823 00:06:45,7.367 00:07:01,8.115 00:07:05,8.361 00:07:19,13.313 00:07:24,7.255 00:07:44,14.877 00:07:49,16.052 00:07:57,5.811 00:08:12,15.962 00:08:20,7.89 00:08:32,4.374 00:08:41,7.653 00:08:43,6.85 00:09:02,5.085