Wikipedia says that the name of concept comes from physics, but I cannot find any similarity between these two concepts.
If you have a linear rod, the center of gravity is the first moment (the expected value), and the moment of rotational inertia about the center of gravity is the variance. (A rod with centrally located mass will have less inertia than a rod with heavy concentrations of mass at the tips.)
Moments gives information about the statistical distribution. We judge one dataset over other based on moments of the dataset (e.g. difference between means(1st moment) of the 2 dataset)