1
$\begingroup$

I have this dataframe where I created the 3rd column using the first two columns. Both ${Y}$ and ${X}$ are independent random variables.

$\bar{Y}$ $\bar{X}$ $\bar{Z} = 100\frac{\bar{X}-\bar{Y}}{\bar{Y}}$
13435 13502 0.4987
21847 22354 2.3207
15584 18014 15.5929
17121 16738 -2.2370
18758 19664 4.8299
18994 22234 17.0580
22864 23555 3.0222
30365 31452 3.5798
29850 34808 16.6097
38674 38791 0.3025
47803 39717 -16.9153
88777 91756 3.3556

When I manually calculate the expectation of $\bar{Z}$ using the data in the 3rd column (in the above table) and pass it statistics.mean() function in Python, I get $E[Z] = 0.04.$

On the other hand, I know that I can calculate $E[Z]$ also as follows:

$E[Z] = E[100\frac{X-Y}{Y}] = 100E[\frac{X}{Y}-1] =100E[\frac{X}{Y}]-100 =100E[X]E[\frac{1}{Y}] -100 $ (as according to this link https://en.wikipedia.org/wiki/Ratio_distribution I can write $E[\frac{X}{Y}] =E[X]E[\frac{1}{Y}]$ )

but then I get $E[Z] = 33.75$ (since the mean of $\bar{X}$ and $\frac{1}{\bar{Y}}$ are $E[\bar{X}] =31048.75 $ and $E[\frac{1}{\bar{Y}}] = 0.000043$, respectively.)

Apparently I get two very different mean values for $\bar{Z}$. When I do the same process for standart deviation, I again get very different values. So which approach should I use? Manually calculating using the 3rd row (by using Python's statistics library mean() and stdev() functions) or using the derivation formula?

In this study http://www.statistics.du.se/essays/D09_Zhang%20Ling%20&%20Han%20Kun.pdf, as far as I understood, they straightaway calculated mean and standard deviation by using the data in Percentage and Absolute columns.

I am really looking forward to some informative answers!

Thank you!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

There are a couple of issues that can be addressed here.

First,

In general, the expected value of the product of two random variables does not need to be equal to the product of their expectation. This means that

$$ E(XY) \neq E(X)E(Y) $$

The equality only holds when the random variables are independent. But... This links directly to the second issue.

Second

Do not mix random variables with samples. The above property holds for independent random variables, but it is very much likely that the samples you display here are actually correlated, so you can't use above property on your samples.

Conclusion

For this reason, the correct result would be to compute the third column and then obtain the mean of that column. Or, if you want to use an alternative derivation, you could go as far as this:

$$E(z) = E(100\frac{x-y}{y}) = 100 E(\frac{x}{y}-\frac{y}{y})=100E(\frac{x}{y})-100$$

But $E(\dfrac{x}{y})\neq E(x)E(\dfrac{1}{y})$

Improve this answer
$\endgroup$
3
  • $\begingroup$ Also to calculate the standard error, again, the best I can do is calculating the 3rd row and computing the standard error of that row? $\endgroup$
    – gülsemin
    Commented Apr 15, 2021 at 18:45
  • $\begingroup$ Thanks for the answer! Yes my samples are highly correlated such that I get 0.986014 and 0.98846 for spearman and pearson coefficient, respectively. So if my two samples were independent then I could always say 𝐸(𝑥𝑦)=𝐸(𝑥)𝐸(1/𝑦). Then which method would give me a better result? $\endgroup$
    – gülsemin
    Commented Apr 15, 2021 at 18:57
  • 1
    $\begingroup$ If the samples were independent then both methods should provide exactly the same result. For this, I recommend you to always use the 3rd column approach and forget about the other way. It keeps things simpler. And yes, you should use the 3rd column also for the standard dev. $\endgroup$
    – Álvaro Méndez Civieta
    Commented Apr 16, 2021 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.