Say we run an experiment and observe the following impact on a variable of interest (one row per experimental unit):

  Observed benefit (%)
1                  20%
2                  60%
3                 180%
4                 -10%
100               214%

where Observed benefit is simply measured as: 100 * (After - Before)/Before (i.e. before and after running our test).

Now, say someone asks us if we should embrace the proposed feature based on the results of the experiment. One way to answer this question is to measure the "expected benefit", i.e. $E(\%)$, we will see using the data we collected, and embrace the feature if it's greater than 0.

I see a potentially dangerous "positive bias" in using a decision rule based on this estimate to decide whether or not to embrace this feature.

Positive variables can never become lower than 0, so the possible smallest drop the variable can experience is -100%. However, positive variables can definitely experience changes much greater than 100%. Can this bias us to accept changes? Or is my fear unfounded?

Examples in finance:

Say we want to test a new trading algorithm that buys and sells securities. We take a sample of securities, buy them and sell them using the new algorithm, and observe a distribution of returns on these securities. On all securities, you can at most lose all the money you invested, i.e. see a return of -1X, but you could see much higher returns (e.g. 2X and beyond).

Given the above, it seems likely that the sample mean estimator (for $E[\text{%}]$), which is not biased per se, will be greater than 0% and potentially bias us to adopt the feature we are testing if we are just trying to maximize expected returns, which is a reasonable goal.

But the above seems problematic to me. One should not always expect positive returns regardless of the feature/strategy we are testing... So what's wrong with this thinking?

Examples in advertising

One is often interested in CPA (cost per "action") or CPC (cost per click). The denominator in both quantities can be relatively small (few actions and few clicks for many dollars spent). So it's common that these quantities reach very high values and thus % can be highly variable.

With this in mind:

  • Am I incorrectly assuming that one may overestimate $E(\%)$ in this problem using sample averages?
  • Given that percentages can reach very high values very easily, should one take "additional" cautionary measures when estimating the significance of a positive result? That is, how do we protect ourselves from overestimating % ? Is the "usual playbook" of methods to avoid overestimating significance enough here, or should one proceed with even additional care?
  • $\begingroup$ You're asking about bias in the sample average of $\text{impact}$ across experimental units, but what population quantity is this average thought to be a biased estimate of? $\endgroup$ – Glen_b Jun 24 '16 at 4:19
  • $\begingroup$ Thanks @Glen_b Not sure I follow, but in the financial example, we could assume that we want to test a new way of buying and selling securities. We choose a sample of securities, and buy them and sell them using the new "strategy" that we are testing. We observe a distribution of returns on these securities, and, say we see that it has a long positive tail.. $\endgroup$ – Amelio Vazquez-Reina Jun 24 '16 at 4:24
  • 1
    $\begingroup$ I don't understand your objection to concluding that E(x) > 0. That's not the value it always has to take, that's just its long-run average. If you're uncomfortable drawing conclusions from an average only, then that's a good thing! You're thinking the right way. But that discomfort isn't specific to variables with a positive skew $\endgroup$ – shadowtalker Jun 25 '16 at 23:12
  • 1
    $\begingroup$ Thanks @ssdecontrol. I mentioned bias (not skew) deliberately since my concern is that my estimates of $E(\%)$ are positive, but the feature under test would really have no positive impact in practice. The more I think about this question, the more I think my fear was/is unfounded, but I wasn't sure initially. $\endgroup$ – Amelio Vazquez-Reina Jun 26 '16 at 16:39
  • 2
    $\begingroup$ @AmelioVazquez-Reina if you think you've answered the question for yourself, I'd encourage you to post your thinking as an answer to your own question here. It will be very helpful to future readers. $\endgroup$ – shadowtalker Jun 26 '16 at 17:06

As mentioned in the OP and comments, the sample mean is an unbiased estimator of the population mean so we should not fear any "positive bias" when using it to obtain a point estimate of the forecasted impact ($\mu$).

That said, if the percentages tend to be highly variable, which may happen if you work with small denominators, you have to factor that in either:

  • by properly reflecting this natural variability in your uncertainty of your estimation of $E(\mu)$ (e.g. via interval estimates or the posterior, which can inform final decisions via e.g. ROPE).
  • or by making sure that the observed result (e.g. $\bar{x}$) is significant enough before embracing it (e.g. NHST).

In the former, you would factor that in through the likelihood or the prior. Alternatively it would come directly from the estimator. If using the latter, the fear of easily embracing a false positive when assessing $\bar{x}$ (because is has a natural high variation), is unfounded since the distribution under the null $H_0$ should have a proper (as in "long enough") positive tail.

Important note:

As discussed in the comments, when working with expectations over time, where the change is applied repeatedly, the sample mean is biased as it does not account for compounding. However, when we are considering single-time changes, the sample mean is unbiased. In the latter, we ask ourselves "What would happen if I apply the same change I tried my experimental units, to other units (once)?".

In other words, with no compounding, the sample mean is unbiased. With compounding it is of course biased and "log returns" or the "geometric mean" are better suited.

| cite | improve this answer | |
  • 2
    $\begingroup$ This is good. I'd like to make two additions: 1) you need to be sure that any NHST you apply is robust to skewed distributions, and indeed many are not; you can't always rely on the central limit theorem. 2) it sounds like you're concerned about the variance of the result -- you might want to consider whether the mean is really the right evaluation criterion, or if something like the median would be better suited. $\endgroup$ – shadowtalker Jul 14 '16 at 2:07
  • $\begingroup$ Thanks @ssdecontrol Interestingly we got another answer to this Q from fcop with claims of bias behind these estimators. $\endgroup$ – Amelio Vazquez-Reina Jul 14 '16 at 12:48
  • $\begingroup$ It just occurred to me that your data consists of rates of return. The arithmetic mean is an incorrect characterization of average return; you ought to be using the geometric mean in this case. $\endgroup$ – shadowtalker Jul 14 '16 at 13:02
  • 1
    $\begingroup$ Thanks. @ssdecontrol I got it now. The problem seems to come from the fact that the sample mean does not account for compounding. This is the case with expectations over time, where we repeat the experiment multiple times. However, some of the problems I am considering are one-off changes. That is, we learn what happened to experimental units, and ask ourselves "What would happen if I apply this strategy to other units?". In other words, with no compounding, the sample mean is unbiased. With compounding it is of course biased and "log returns" or the "geometric mean" are better. $\endgroup$ – Amelio Vazquez-Reina Jul 14 '16 at 20:43
  • 1
    $\begingroup$ @AmelioVazquez-Reina yes exactly $\endgroup$ – shadowtalker Jul 14 '16 at 21:08

I think it depends on what you want to do. Looking at your example from finance, it seems to me as if you want to estimate your total return (in euro or dollar) for a portfolio, using an estimated percentage return on a sample. Let's say you have a sample of securities with values $v_i$, and returns $r_i$, $i=1, 2, \dots n$, so the rates of return are $rr_i=\frac{r_i}{v_i}$. This is a sample of your whole portfolio of size $N$.

The total value of your portfolio is $V=\sum_{i=1}^N v_i$. The total return of the portfolio (in euro or dollar) is $R=\sum_{i=1}^N r_i$.

Your goal is to estimate the rate of return of the whole portfolio $\frac{R}{V}$, using only information from the sample.

In this case you can estimate the rate of return $\widehat{rr}$ from the sample in two different ways:

  1. Mean of the returns in the sample : $\widehat{rr}^{(1)}=\frac{1}{n} \sum_{i=1}^n rr_i$ (this is what you do in the example)
  2. the second option is to use $\widehat{rr}^{(2)}=\frac{\bar{r}}{\bar{v}} $, (where $\bar{r}=\frac{1}{n} \sum_{i=1}^n r_i$ and $\bar{v}=\frac{1}{n} \sum_{i=1}^n v_i$) or the ratio of the average return (in euro) in the sample and the average value in the sample.

It can be shown that both estimators are biased estimators for $\frac{R}{V}$, but that the bias of the second one is smaller (see this link)

So the less biased estimate of the return of your portfolio is $\hat{R}=\frac{\bar{r}}{\bar{v}} V$.

The one you talk about is the first one, so the one with the highest bias.

There is a whole theory on this, you should google for ''ratio estimator'' or ''ratio of means versus mean of ratios''.

I found an example at this link

EDIT because of your comment below:

first of all, you asked for a reference, follow this link

the sample average $\frac{1}{n} \sum_{i=1}^n rr_i$ is an unbiased estimator of the population mean ratio $\frac{1}{N} \sum_{i=1}^N rr_i$ (note the $n$ for the sample and $N$ for the population).

However, and as I said at the beginning of my answer, it depends on what you want to do.

(a) if you want to estimate the mean population ratio, then you can use the sample average,

(b) but you seem to look for a rate of return that you can estimate from your sample and that you want to use to estimate the rate of return for the whole portfolio. If you know the whole portfolio then its total value is $V=\sum_{i=1}^N v_i$ and its total (euro-) return $R=\sum_{i=1}^N r_i$. Note that I use $N$ so it is about the population. This means also that the population mean value is $\mu_v = \frac{1}{N} \sum_{i=1}^N v_i$ and the population mean return (in euro) is $\mu_r = \frac{1}{N} \sum_{i=1}^N r_i$.

On the other hand, the rate of return for the population is $\frac{R}{V}=\frac{\sum_{i=1}^N r_i}{\sum_{i=1}^N v_i}$ which is obviously equal to $\frac{\frac{1}{N}\sum_{i=1}^N r_i}{\frac{1}{N}\sum_{i=1}^N v_i}$ which is equal to $\frac{\mu_r}{\mu_v}$, or the population rate of return is equal to the population mean return in euro divided by the population mean value. Hence the idea to use $\frac{\bar{r}}{\bar{v}}$ as an estimator for the population rate of return.

So if you want to estimate the rate of return of the whole portfolio then it is better to use the ratio of the averages.

If you want to estimate the mean of the individual securities' rate of return then it is better to use the average of the returns.

| cite | improve this answer | |
  • $\begingroup$ Thanks @fcop. This is helpful. Do you know of any sources behind the claim that "it can be shown that both estimators are biased"? This statement seems to contradict what's in the comments and the other answer I wrote. I'm interested to see how they reach that conclusion. $\endgroup$ – Amelio Vazquez-Reina Jul 14 '16 at 12:45
  • $\begingroup$ @Amelio Vazquez-Reina: I certainly have references, I will look it up but I don't have the time today. About the unibiasedness and the other comments: I will add it at the end of my answer. $\endgroup$ – user83346 Jul 14 '16 at 12:54
  • $\begingroup$ @AmelioVazquez-Reina it's because each data point is a ratio, and it has nothing to do with skew in general. This answer is saying that the sample mean of ratios is in fact biased. $\endgroup$ – shadowtalker Jul 14 '16 at 12:56
  • 1
    $\begingroup$ However this answer misunderstands the question; it is not about a sample of securities, it is about estimating the return of a single security over time. $\endgroup$ – shadowtalker Jul 14 '16 at 13:00
  • $\begingroup$ @ssdecontrol: I did not say that the sample mean of ratios is biased, in fact a sentence like ''the sample mean of ratios is biased'' makes even no sense. You should first say what you want to estimate. So you can say that ''the sample mean of ratios is a biased biased estimator FOR ...'' and you have to fill in the points. This is why I start with 'it depends on what you want to estimate'. In the OP's case he wants to estimate the rate of return for a protfolio. $\endgroup$ – user83346 Jul 14 '16 at 13:22

Asymmetry of percentage/proportion changes: The issue you have raised in your question is not actually a statistical problem. Rather, it is a mathematical problem about the appropriate way to measure percentage changes in a non-negative quantity. If you want to aggregate positive and negative percentage changes for a non-negative variable, you need to measure these positive and negative changes in such a way that they are considered to be of "equal magnitude" when they would cancel out with each other in aggregate.

We known that this does not happen if we use the percentage/proportion change as the measure, since this measure is "asymmetric". To see this, suppose we have changes measured by:

$$r_{t+1} \equiv \frac{a_{t+1} - a_t}{a_t}.$$

Over a sequence of time periods $t=1,...,T$ the final amount we obtain can be written in terms of the initial amount, and these proportionate-change measures, as:

$$a_T = a_0 \prod_{t=1}^{T} (1+r_t).$$

It is easy to see that if we take an initial quantity, and impose an increase and a decrease of the same "magnitude" (i.e., $r_1 = r$ and $r_2 = -r$) then we don't get back to the value we started from. Instead we have the lesser amount:

$$a_2 = a_0 (1+r_1)(1-r_2) = a_0 (1+r)(1-r) = a_0 (1-r^2).$$

Hence, we can see here that positive and negative percentage/proportion changes are asymmetric, in the sense that the negative proportionate change is of greater "real" magnitude than a positive proportionate change using the same value. You are right to be concerned that negative changes are of greater force than positive changes with the same headline percentage value.

How to measure percentage/proportion changes: Because of the above phenomenon, when we are dealing with data showing percentage/proportion changes in a quantity, we should measure the magnitude of the changes in a way that ensures that positive and negative changes of the same "magnitude" cancel each other out. This is accomplished by measuring change on a logarithmic scale, using the measure:

$$\delta_{t+1} \equiv \frac{\ln a_{t+1} - \ln a_{t}}{\ln a_{t}}.$$

(It is worth noting that this measure is related to the proportionate change measure by the equation $\exp (\delta_{t+1}) = 1 + r_{t+1}$.) In finance, this quantity is called the force-of-interest.$^\dagger$ Over a sequence of time periods $t=1,...,T$ the final amount we obtain can be written in terms of the initial amount, and these force-of-interest measures, as:

$$a_T = a_0 \exp \Big( \sum_{t=1}^T \delta_t \Big).$$

Unlike percentage changes on the raw scale, on this scale, equal positive and negative values of the force-of-interest cancel out, leaving you with the value you started with. (This is easily seen from the fact that these values enter the above equation through their sum.)

How to avoid "bias" in your data analysis: From the above exposition, we can see that all you need to do to deal with this issue is to measure changes on the logarithmic scale, using the force-of-interest. Convert your percentage changes to measures of the force-of-interest and then you will be using a measure that is "symmetric", in the sense that positive and negative changes of the same value cancel out with one another.

As other commentators have pointed out, this issue is entirely distinct from the issue of statistical bias in your estimators. You will still need to choose an appropriate statistical analysis to model your data, and you should choose appropriate statistical estimators. The advice here will give you a symmetric measure for changes in your variable. If you combine this with an unbiased estimator of the true mean force-of-interest then you will have a reasonable measure of the tendency of your experiment to positively or negatively impact your variable of interest.

$^\dagger$ It is worth noting that the force-of-interest is usually defined on a continuous scale, but here we are looking at the analogy over a discrete time scale. For a non-negative accumulation function $a$, taken over continuous time, the force-of-interest is given by $\delta(t) = \frac{d}{dt} \ln a(t)$.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.