# Statistical Approaches for Interpreting Weekly Changes in a Percentage Metric

## Overview

Each week we have some number of trials and successes, and we track the percentage of successes each week, this is our metric.

Made up example: A loan company - each week we get some applications, and make some rejections. Different factors influence the number of applications and the acceptance rate (marketing campaigns, changes to credit scoring models etc).

If $$p_i$$ is the proportion accepted in week $$i$$, then it may happen that a statistical significance test in proportions on $$p_i$$ , $$p_j$$ concludes a difference. However, this isn't really what I am interested in, because for my process maybe it is usual(common) for that to happen. (See example below). I am really interested in the distribution of the $$p_i$$s - is the $$p_i$$ I have this week very different to the $$p_i$$s I have seen before.

The business problem I am trying to solve: I have seen colleagues "interpreting" changes in the metric week to week. Celebrating if one week the metric goes from 90 to 92 and then the next weeks panicking if it drops down to 90, 89 - whereas maybe this metric always just hovers around 90%. I would like to be able to say, with some statistical backing, that we expect fluctuations like this in the metric and it is not a cause for concern or celebration, and on the other hand would like to be able to know at which point we should be investigating further!

## Numerical Example

Data points [92. 87. 90. 91. 90. 90. 88. 88. 90. 92. 91. 88. 85. 93. 92. 92. 89. 88. 88. 90. 90. 89. 89. 90. 92. 91. 87. 90. 90. 87.]

Each week we have thousands of applications (trials) - the exact number varies week to week. If I look at week 3 and week 4 the metric has the value of 90 and 91 respectively. Say for example we had 10,000 applications in both of those weeks. Then, if I did a z-test for difference in proportions between these two weeks we would get a statistically significant difference (at the $$\alpha=0.05$$ level). With the values above, and having this many applications, I imagine this could be the case between most weeks and I don't think this is really the question I am interested in.

What I am interested in is that all the values hover around 90% with some deviation.

What I would like to do is look at the average value of the metric and standard deviation, and then plot upper and lower lines some number of std deviations from the mean. I suppose I am really trying to model the $$p_i$$s but am not sure how to do that. I don't think I can use a normal distribution, because each $$p_i$$ is a proportion, it cannot be outside the range 0-1. If I modelled the $$p_i$$ with a normal distribution then it could happen that I get an upper or lower threshold outside the range of 0-1.

How can I approach this problem? I am even having trouble describing my problem so knowing the correct terminology would be helpful too.

Note the example use case and data points are fictional. Perhaps another way to think of this is that each week I am tossing a coin thousands of times. It may be week to week I am tossing two different coins, so get a statistically significant difference in the proportion of heads, but I don't care about this, I care that all these coins have come from the same machine.

• one simple thing you could do is take the logit of these probabilities and then calculate the mean and standard deviation (and assume/test whether they follow a normal distribution) Commented Sep 5, 2023 at 13:52
• are you expecting any time series pattern? or do you assume that each week is IID Commented Sep 5, 2023 at 13:55
• Another simple thing to do would be to look at the variations per day. If there are 10k applications in a day, then you can split them up into groups and calculate percentages as for the days. If these inter-day percentages vary substantially, then it indicates that your weekly variation is not very meaningful. Commented Sep 5, 2023 at 20:23
• @seanv507 in reality there probably is a time series pattern but I would also be happy hearing simpler approaches that don’t take that into account. Commented Sep 6, 2023 at 20:39
• @picky_porpoise thank you for your comment. I am sure I am just being slow but I am having trouble understanding this suggestion. Would you be able to explain it another way or with an example? Commented Sep 6, 2023 at 20:41

1. transform your $$p_i$$ into $$z_i=\text{logit} (p_i)=\frac{\ln(p_i)}{\ln(1-p_i)}$$. these are distributed over the real line so could be fitted by a normal distribution (or other common distribution).
2. estimate the distribution of $$z_i$$, (plot histogram and/or fit a distribution eg normal, by calculating the mean and standard deviation of the $$z_i$$)