4
$\begingroup$

I am kind of new to statistics so this might have been asked before (and I either haven't found or understood it).

Given a dataset of fractions: $[a_1/b_1, a_2/b_2, .. a_n/b_n]$, the overall ratio of the dataset would be calculated like this: $ \frac{\sum_{i=0}^n a_i}{\sum_{i=0}^n b_i}$ (1) - (not sure if it would be called "mean" in this case, because it is not the mean of the elements in the dataset).

I am interested in calculating the required sample size (based on the sum of $b$'s) that is required to detect a difference of $x$ percent (comparing the result of (1) from a population with the result of (1) from a sample) with a confidence of $y$. $x$ and $y$ are input variables.

Looking at my data, I already figured out that neither $a$ nor $b$ on their own follow the normal distribution (using the Shapiro-Wilk test), so I don't think I can use the standard formula: $n=(\frac{Zσ}{E})^2$.

For context an example: A datapoint would consist of $a$, the purchase amount and $b$, the number of purchases, but the data is pre-aggregated. E.g. 3 purchases would generate $5 (but we don't know how the $5 are distributed among the three purchases, it could even involve one or more "free" purchases).

More context: I would like to know what the minimum number of purchases is, that I need to be able to reliably detect a change in the value/purchases ratio. Each datapoint (fraction) has a timestamp.

Happy to explain more if it is not clear what I am trying to achieve.

$\endgroup$
13
  • $\begingroup$ I'm not sure why you are interested in the sum of ratios, instead of, for example, the sum of $a_1, ..., a_n$ divided by the sum of $b_1, ..., b_n$. Is it an error of notation, or are you really interested in the sum of ratios? If so, why? In general, it would help if you could give more information about the question you're trying to answer ultimately (can you articulate it with as few statistical terms as possible?), and about your dataset (e.g. what does each element of your list of ratios refer to? To a single product? Can a product appear multiple times in the list? Is it a time series?). $\endgroup$
    – J-J-J
    Aug 29, 2023 at 9:12
  • 1
    $\begingroup$ Yes, sorry I noticed the error in the notation earlier today, but your comment was quicker than my fix ;) $\endgroup$
    – kev
    Aug 29, 2023 at 10:45
  • 1
    $\begingroup$ Added more context as well. $\endgroup$
    – kev
    Aug 29, 2023 at 11:24
  • 1
    $\begingroup$ So each transaction (which may include multiple items) has a average value, you want to test whether some manipulation (experiment) will increase the average value per transaction, and you need figure out what sort of sample size you need for the experiment, is that right? A lot of this will depend on how what effect sizes you're interested in. If you want to detect even a very small effect, you'll need a huge sample. So what's the minimum effect size you'd care about? $\endgroup$
    – num_39
    Aug 31, 2023 at 11:00
  • 1
    $\begingroup$ @num_39 That's more or less what I meant when I said "A general answer might still be possible without defining a specific value for the effect size" (even though the solution is not necessarily under the form of a formula). However, without more details about the dataset, we might not be able to suggest a correct test in the first place (e.g. some of the questions I asked earlier have not been really clarified, in particular if different ratios can refer to the same product at different points in times; generally what are $a_n$ and $b_n$ exactly.) $\endgroup$
    – J-J-J
    Sep 4, 2023 at 11:09

1 Answer 1

1
$\begingroup$

Here is a proposal. Keep track of two rolling windows:

  1. A large window $W_1$ using the $a$'s and $b$'s to define what is standard behavior.
  2. A sub-window $W_2 \subset W_1$, for telling when an anomaly has occurred.

Both windows should end at the present, and their size should be determined by the sum of $b$'s, rather than time.

There are many ways to learn the distribution using $W_1$. A straightforward way is bootstrapping. Resample the data in $W_1$ n_sample times, getting a variety of fractions $\Sigma a / \Sigma b$. Using a confidence level $\alpha$, sort the resulting fractions, and declare anything in the percentile range $(\alpha/2, 1-\alpha/2)$ as "typical".

Then for $W_2$, compute a single ground truth fraction $\Sigma a / \Sigma b$, find where it fits in the sorted array from $W_1$, and declare an anomaly if it's below the $(\alpha/2)$th percentile or above the $(1-\alpha/2)$th percentile.

The hyperparameters here are

  1. $b_1$ the sum of most recent $b$'s defining $W_1$.
  2. $b_2$ the sum of most recent $b$'s defining $W_2$ ($b_2 \leq b_1$).
  3. n_samples the number of bootstrapping iterations for getting the distribution of fractions.
  4. The confidence level $\alpha$.

You could try running different parameter configurations and see how often anomalies are detected.

$\endgroup$
1
  • $\begingroup$ Interesting approach (as far as I understand). I will tree and see how far I can get. $\endgroup$
    – kev
    Oct 16, 2023 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.