7
$\begingroup$

My data is some percentages on how many transactions from a whole are missed each month. The percentages are for 13 months and ranging from 97 to 99 percent. I was asked to calculate the mean and standard deviation, but I am unsure if the result would be meaningful and if the mean could be calculated for percentages in the traditional sense (e.g., in Excel doing AVERAGE(percent array)) vs. some other method (weighted averaged) since I do not have any other information other than the percentages.

Please help me to understand

  1. If the mean and standard deviation for percentages can be calculated,
  2. What conditions are needed in order to calculate the mean and standard deviation for percentages,
  3. Alternatives determining spread and central tendency for percentages over a period of time.

For example, would the following be incorrect to do assuming that the percentages represent # transactions missed / # total transactions for each different months with the same calculation being used for each month:

Month   Data
Feb-15  98.0%
Mar-15  98.7%
Apr-15  97.0%
May-15  99.9%
Jun-15  98.7%
Jul-15  97.9%

Mean
98.4%

SD (Population)
0.90%

From this post is seems like it should be done with weighted averages since the totals are different, and that calculating the mean and standard deviation is possible for percentages only if they come from the same total, meaning the above would be incorrect and that I would need additional information to determine the weight to multiple the percentages by, if that is correct.

Improve this question
$\endgroup$
5
  • 6
    $\begingroup$ Of course you can calculate mean and SD of percentages: just plug them into the formula. What should be of concern is how the resulting values will be interpreted. Could you explain why you have been asked to perform this calculation and what will be done with the results? $\endgroup$
    – whuber ♦
    Commented Oct 30, 2015 at 2:35
  • 3
    $\begingroup$ @whuber is right here. After all, a percentage is just a mean of 0's and 1's. What is most important is not whether a mean and standard deviation can be calculated, but what you want to do with them and if it makes sense in that context. $\endgroup$
    – StatsStudent
    Commented Oct 30, 2015 at 2:41
  • $\begingroup$ It was a blanket request to calculate the mean and standard deviation of a set of metrics to set thresholds for them. So in this example, the 98.4% average would be used to measure the current month against, with the assumption that the more standard deviations away it is from the mean, the more of a concern it would be. $\endgroup$
    – daneshjai
    Commented Oct 30, 2015 at 5:02
  • 2
    $\begingroup$ As you can see, two standard deviations away from the mean is already in a territory of over 100%. If your task is to see how extreme a new observation is as compared to the historical data, you could consider using some rank statistic. E.g. just see how many months out of the total number of months have been worse or better than the current month. You will get scores like 2/13 or 1/13, or even 0/13. $\endgroup$
    – Richard Hardy
    Commented Oct 30, 2015 at 11:05
  • $\begingroup$ @RichardHardy Of course, with ranks some value is always the highest, and some the smallest, so it is not immediately apparent how ranks or rank statistics will provide a solution to "is this value different enough to be of concern?" $\endgroup$
    – Alexis
    Commented Nov 15, 2019 at 16:22

3 Answers 3

Reset to default
4
$\begingroup$

As others have pointed out, whether it is correct to calculate the mean and the standard deviation of percentages depends on your intended use. For your use, at least as I understand it, it seems to be incorrect.

As I understand from your question and comment, you are trying to do anomaly detection. You are basically asking:

Is the number of missed transactions within what could be considered "normal", or does it deviate so much to be considered anomalous?

There is no clear-cut answer to that question. The best you can do is to calculate the probability:

Assuming a known probability of a transaction to be a "mis", how probable is to have the given number of misses in a month?

If it is very improbable (say, probability below 0.05), you may consider it to be anomalous. So the question remains how to calculate this probability. If your percentages were normally distributed, you could easily derive it from the mean and the standard deviation: values that are more than 2 SDs away from the mean appear with probability below 0.05. That's presumably the reason why you were asked to compute these values.

However, your percentages are not normally distributed! As Richard Hardy pointed out in his comment, two SDs above the mean are already impossible to achieve, as it would be above 100%. You need to use a different, more appropriate probability distribution. Without having further domain knowledge of your data, the best you can do is to use the binomial distribution:

$$ P(k) = {n \choose k} p^k (1-p)^{n-k} $$

with $n$ being the number of transactions and $k$ the number of misses in the month in question. You can estimate $p$ from historical data, as the fraction of the total number of misses and the total number of transactions in the past months.

Having all this, you can calculate the cumulative probability of observing at least as many misses as you actually had in the month in question. If that probability is below some pre-defined level (for example the above mentioned 0.05), you'd consider it an anomaly.

For completeness: If you want to be even more precise (which I doubt, considering that you were given a wrong task in the first place), you can get a confidence interval of $p$ by modelling it by the beta distribution, and use the extreme, but still plausible $p$ in the above binomial distribution. The parameters of the beta distribution would be e.g. $\alpha = $ (the number of misses) and $\beta = $ (total number of transactions $-$ the number of misses).

Improve this answer
$\endgroup$
1
$\begingroup$

As the article linked in the question states, you should not calculate the average of percentages using the same method for whole numbers.

You must use a weighted average.

See this recent article has more details https://www.indeed.com/career-advice/career-development/how-to-calculate-average-percentage but it explains the same method as the article linked in the question.

Improve this answer
$\endgroup$
-3
$\begingroup$

I do not like doing those calculations with percentages. First option is to work with the numerators and denominators, and then do some manipulation. Second option is to convert the percentages into log values, which will force results into the 0 to 100 percent range.

Improve this answer
$\endgroup$
3
  • $\begingroup$ This does not appear to be answering the question. $\endgroup$
    – Michael R. Chernick
    Commented Nov 15, 2019 at 16:25
  • 1
    $\begingroup$ The second option makes no sense. If you have percentages between 0 and 100 then taking logs won't work if you have 0 values and it forces percentages below 1 to below 0. Occasionally there is some point to taking logarithms of percentages, but typically as an approximation to working on logit scale. $\endgroup$
    – Nick Cox
    Commented Nov 15, 2019 at 17:19
  • $\begingroup$ I am used to working with probabilities, and almost exclusively natural logarithms (the logit scale). It works in that domain, with certain assumptions. I should have qualified my comment. Initially, I thought it could be applied more generally, but you are right, it cannot. $\endgroup$
    – Raymond Anderson
    Commented Nov 16, 2019 at 8:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.