Timeline for Can I use 'mean ± SD' for non-negative data when SD is higher than mean?
Current License: CC BY-SA 4.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 16, 2023 at 16:20 | comment | added | OverLordGoldDragon | On second thought the handling is okay, the issue's related to SD itself. This answer's nice for inheriting SD properties exactly. I proposed something that should be more robust but didn't test every angle. | |
| May 15, 2023 at 12:01 | comment | added | OverLordGoldDragon | I re-evaluated events in context of an active network, and found I went overboard. In the network I frequent, DSP.SE, the norms are quite different. I should've raised my concerns more politely. Sorry about that @ IgorF. @Firebug (Also flags and mods have nothing to do with my comment.) | |
| May 15, 2023 at 11:09 | comment | added | Scortchi♦ | Oh I see! So you're not claiming that $ \frac{1}{N_A - 0.5} \sum_{i:x_i \gt \overline x} (x_i - \overline x)^2 $ is an unbiased estimator of $\operatorname{E}[(X- \operatorname{E} X)^2|X>\operatorname{E}X]$ (which I don't suppose is the case, in general). | |
| May 15, 2023 at 10:50 | comment | added | Igor F. | @Scortchi-ReinstateMonica: See my comment to Firebug, 2023-05-12 13:59:16Z. It's easy to see if you go over variances. Due to the decomposition, $(N-1) V = w_A V_A + w_B V_B$ (I take (N-1) for the unbiased estimate). For perfectly symmetric data, $N_A = N_B = N/2$, and we require $V_A = V_B = V$ and $w_A = w_B$. So it must be that $w_A = w_B = (N-1)/2 = N_A - 0.5 = N_B - 0.5$. | |
| May 15, 2023 at 10:12 | comment | added | Scortchi♦ | @IgorF. How did you derive the correction of -0.5? | |
| May 14, 2023 at 11:03 | comment | added | OverLordGoldDragon |
I don't think you've handled it ideally either. Try [-2, -1, -1, 0, 1, 1, 2] and replace 0 with 1e-15. There's no meaningful difference between the two, yet your metric suggests otherwise, which is an instability.
|
|
| May 12, 2023 at 13:59 | comment | added | Igor F. | @Firebug I updated the formulae to satisfy the requirement: $sd = sd_A = sd_B$ for perfectly symmetric data. It turns out, the correction is not $-1$, but $-0.5$ (plus a correction term if any $x_i = \overline x$). | |
| May 12, 2023 at 13:56 | history | edited | Igor F. | CC BY-SA 4.0 |
Corrected formulae, refactored code
|
| May 12, 2023 at 7:45 | history | edited | Igor F. | CC BY-SA 4.0 |
improved notation
|
| May 12, 2023 at 7:42 | comment | added | Igor F. | @Firebug Thanks. I admit that my introduction of the Bessel's correction in the decomposed $sd$ was just copy-paste from the ordinary $sd$ and I have no idea whether it's justified. I'm curious whether you (or anyone else) have any comments about it. | |
| May 12, 2023 at 6:57 | comment | added | Firebug | Quite interesting proposal. It would be interesting to know the asymptotic properties of these, and perhaps derive better estimators, but I like the base idea (since they still recover the original standard deviation and thus do not suffer from some of the other caveats in other answes) | |
| May 11, 2023 at 13:31 | history | answered | Igor F. | CC BY-SA 4.0 |