I have a set of pipes that are supposed to receive balanced load from a source. For instance, with 100 of inputs if I have 5 pipes each is supposed to handle 20. But that's not the case in reality and normally it's like:

[23, 26, 19, 18, 14]

I'm working on a system to trigger an alarm if one of the pipes are underloaded or overloaded. So clearly I'm looking for outliers. I found that standard deviation would be good fit for detecting that but I wonder that the mean of random numbers might skew the expected value (here 20).

So the question is, if instead of the mean I use 20 and calculate the SD based on that, how correct the final result will be? Or am I completely confused?

Few cases that might trigger the alarm:

[20, 20, 20, 3, 37]
[20, 0, 20, 20, 40]
  • $\begingroup$ I see that the numbers don't add to 100 even though the average is supposed to be 20. What would you want to do if they all said, say 13? (in other words, are you interested in deviation from a specified value, or just finding unusual values among what you have? Can you show some cases (loads on pipes) you'd want to trigger an alarm for and others you wouldn't? $\endgroup$ – Glen_b Aug 5 '15 at 6:51
  • $\begingroup$ I see you edited it so it now does add to 100. Will it always add to exactly 100? $\endgroup$ – Glen_b Aug 5 '15 at 6:54
  • $\begingroup$ @Glen_b, I corrected the numbers. They should add up to 100. I'll add the trigger cases in minute. Yes, they always add up to 100. $\endgroup$ – norbertpy Aug 5 '15 at 6:54
  • $\begingroup$ If the numbers always add to 100, it doesn't matter whether you use a specified mean of 20 or the sample mean (which must also be 20). Of course if you cared about estimating the sd, it would change the denominator, but that's not the exercise here. Indeed, I wonder why you'd need the sd at all. Would it not make sense to specify a value for "too high a load" and one for "too low a load" and trigger those? Or is two pipes at 27 about as much a problem as one pipe at 30? $\endgroup$ – Glen_b Aug 5 '15 at 6:57
  • $\begingroup$ @Glen_b, I don't have a specific threshold to set but rather I'm interested in seeing that all the pipes are sharing about the same resources. So, [10, 23, 23, 22, 22] would be OK but not the [10, 30, 20, 20, 20]. Sorry if it sounds stupid. $\endgroup$ – norbertpy Aug 5 '15 at 7:03

To answer the question itself:

So the question is, if instead of the mean I use 20 and calculate the SD based on that, how correct the final result will be?

Since the total flow is always 100, the sample mean will always be 20. As a result, if you do the same standard deviation calculation it will make no difference whether you use the known mean or the sample mean.

What might change is the denominator (if you cared about an unbiased-for-variance estimate for some reason), but if you're just comparing with some yet-to-be-determined cut-off there's no particular reason to bother about that. Even if you did divide by $n=5$ rather than $n-1=4$ to make the variance unbiased, all you'd need to do is adjust the cut-off point the same way and performance would again be identical either way. The numerator of the variance is the thing that is putting a partial order on sets of pipe-loads, the rest is really just scaling. Indeed since the mean is fixed one could just as well simply use the sum of squares of the loads as a criterion (that statistic has been used in similar contexts where uniformity is an issue).

I think this particular issue is about as not-really-an-issue as they come. If the standard deviation really does what you want, just think of it as whichever of the two calculations is easiest for you to think about, since it makes no difference. Once you're sure sd does what you want, the next trick is to figure out a suitable cutoff value for your alarm.

  • $\begingroup$ Thank you. I figured out that with total of 100 the mean is always fixed. I was confused. This is good explanation. $\endgroup$ – norbertpy Aug 6 '15 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.