0
$\begingroup$

I have around 8 billion data points, and I need to calculate the distribution and the cumulants of this distribution.

However, due to technical restrictions, and time constraints, I can only calculate those cumulants just for a half of the data, but I still need the cumulants of the whole data points.

Question:

if I have a two distributions and I know their cumulants separately, what is the cumulant of the whole combined distributions in terms of the cumulants of each separate distribution ?

Apart from analytical results, I would also accept approximate results.

Improve this question
$\endgroup$
10
  • $\begingroup$ Which cumulant are you calculating? $\kappa_1, \kappa_2, \kappa_3, ...\kappa_{n-1}, \kappa_n $? $\endgroup$
    – user158565
    Commented Jul 27, 2019 at 16:17
  • $\begingroup$ Hopefully only skewness and kurtosis, but it would be great if someone could give a general method of how to derive the higher order cumulants. $\endgroup$
    – Our
    Commented Jul 27, 2019 at 19:40
  • $\begingroup$ If you want skewness and kurtosis, why you do not calculate them directly, instead of calculating the cumulants and deriving them from cumulants? $\endgroup$
    – user158565
    Commented Jul 27, 2019 at 19:46
  • $\begingroup$ @user158565 Do you know how much space does 8 bilion double-precision data points occupy in memory ? $\endgroup$
    – Our
    Commented Jul 27, 2019 at 19:49
  • 1
    $\begingroup$ Alternatively you could ask for "online" updating formulas for cumulants, analogous to [Online estimation of variance with limited memory ](stats.stackexchange.com/questions/235129/…) and look at the tag online. Also this arXiv paper which tackles a more general problem---cumulant tensors $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jul 27, 2019 at 20:41

1 Answer 1

Reset to default
0
$\begingroup$

Let's say $X_1,X_2$ are independent random variables drawn from the two half-distributions, and $k\sim Bernoulli(0.5)$ is another independent r.v. Then you want to find the distribution of $X=kX_1+(1-k)X_2$.

If you can use cumulants $\alpha_i$ to approximate $\log E(e^{tX_1})\approx\sum_{i=1}^n\alpha_i\frac{t^i}{i!}$ and similar for $X_2$ with your cumulants $\beta_i$, then you can plug those into $$\log E(e^{tX})=\log(.5E(e^{tX_1})+.5E(e^{tX_2}))$$ and differentiate the RHS to get your cumulants for $X$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.