Is coefficient of variation appropriate here?

Question

I am in charge of a large number (100+) of batch computing processes and I am looking for a way to quickly identify which of the processes are 'healthy'.

The processes run every day and a duration is logged. There are times when the processes run longer than expected due to some issue with the process. The more frequently a process runs longer than expected, the less 'healthy' it is.

The expected run time of the processes vary greatly, some have an expected run time of under a minute while others may run normally for up to 8 hours.

Am I correct in assuming I can use the coefficient of variation of the duration of each process to compare the 'health' of one process to another, despite having two different mean durations?

Answer 1

The coefficient of variation is the ratio between the standard deviation and the mean. (Often people like to report this in percent terms, but that is just a convention.) When you have several high values in a sample, they will drive up both the standard deviation and the mean, i.e. both the numerator and the denominator; it's hard to predict in advance which effect is more important in general. It could be done with a precise model for your data, if only by simulation. Thus I wouldn't recommend the coefficient of variation as diagnostic of process health, as it can vary quixotically.

For more detail on the coefficient of variation and its uses and abuses, see other threads here such as How to interpret the coefficient of variation?

I would add various general comments:

1. The coefficient of variation is clearly a ratio, and ratios are often very unstable. Even with your data, which I take to be all positive times, then the instability could still be awkward. Otherwise put, a good estimate of the coefficient of variation can be difficult to obtain, particularly in situations with stretched-out tails, which are precisely where it may be most tempting.

2. It's usually unwise to lean too heavily on any one summary measure. Sometimes you may need a tailored measure, e.g. the fraction of times beyond a threshold measure that is regarded as seriously or strongly problematic. The test of this is what can be constructed from the coefficient of variation alone, namely essentially nothing without other assumptions.

3. The use of the coefficient of variation usually implies that you are better working on a logarithmic scale, or more generally that a transformation might be helpful. If you work on a scale on which your distributions are approximately symmetric, then comparisons are often easier.

4. The coefficient of variation is a natural measure for some named distributions, such as some gammas, some lognormals, and some Weibulls. If you have grounds for regarding any of those as plausible, then it might be helpful.

Posting some of your data if possible might allow more specific suggestions.

Answer 2

Ans.: Don't think so, although coefficient of variation is an indication of relative distance (herein in duration) in time, the problem actually depends on what process is considered "healthy" and what that means anyway.

There are related concepts like robust models and ill-posed problems that are better defined. Consider that if one chooses starting values close to the solutions values the process usually is faster. Some regression methods, e.g., random search, usually take much longer to complete than others, e.g., Simplex. Also, asking a better regression "question" works wonders; Better questions often are faster to completion, for example, frequently reparameterization of the problem works faster, and yields better precision and accuracy. Related to this is that some models are hopelessly unscientific, as the units do not balance or the physics is contradicted, even if the math looks OK. In those cases, constructing models that make more physical sense leads to faster solutions, that is, if everything else on the list here is done properly, including the following consideration; the problem may be ill-posed, in which case, regression for ill-posed problems, e.g., Tikhonov regularization may be much more robust (and thus faster).

Last but not least are programming considerations. For example, using too few decimal places can destabilize a problem causing it to go into a very long or even infinite loop. Code often needs debugging for speed, the first version can take 24 hours to run, and the last 24 milliseconds. Solutions can be local optima, and not provide the correct answer regardless of speed. Fast but wrong can give a false impression of robustness. For example, some solutions may only exist in the complex domain, and unless the programming allows for those solutions, the problem will be solved incorrectly in a local real-valued minimum. In the latter case, unless the complex solutions make physical sense, and they frequently do not make sense, the model is probably the wrong one for the physical circumstances and should be swapped out for something better.

The moral of the story, if there is one at all, is to always check that the program actually converges to the requested precision, which is frequently 1/2 of the number of digits contained in the working precision. If it is not converged, then YOU did something wrong. Models that do not converge are not models, and if not converged, one has to change the code.