Calculating  sample mean and sample variance on all samples vs distinct subsets Consider 1000 samples drawn from an unknown distribution. Whats the difference between the following two ways of calculating sample mean and sample variance?


*

*find sample mean and sample variance over all 1000 samples.

*find sample mean over first 500 samples and sample variance over remaining 500 samples.
What is the difference and which method is preferred?
 A: The first method is preferred as you are using all the information in the sample to get more 'accurate' measures of central tendency (i.e., the sample mean) and the spread (i.e., the variance) in your data.
More formally, if $x_i$ is from an unknown distribution but are iid with finite mean and variance then it follows from the central limit theorem that:
$$\bar{x} \sim N(\mu,\frac{\sigma^2}{n})$$
Informally the above suggests the following two 'facts':


*

*The sample mean is an unbiased estimator of the true mean as the distribution of the sample mean is centered at the true mean and

*The spread of the distribution is narrower with higher $n$. 
Fact 1 suggests that the sample mean will remain unbiased with a smaller sample size (i.e., using 500 instead of 1000 should be fine if you are concerned if the sample mean is biased or not) but fact 2 suggests that the distribution of $\bar{x}$ is 'narrower' with higher $n$ and hence it makes sense to use the full sample of 1000 data points rather than 500 data points. Intuitively, facts 1 and 2 together indicate that the higher the sample size used to compute the sample mean, the lower chances that the sample mean is 'far away' from the true mean.
A similar argument for the sample variance can be made. See the wiki on the distribution of the sample variance. In particular see the formula for $E(s^2)$ (i.e., the mean of the sample variance) and  $\text{V}(s^2)$ (i.e., the variance of the sample variance).
A: If the data are normal, the sample mean and the sample variance are independent. If the data are not normal, the covariance/correlation between the sample mean and the sample variance are $O( \kappa_3 n^{-1})$ where $\kappa_3$ is the (central) skewness of the distribution. If, for any reason, you really need the sample mean and the sample variance to be independent, then you can calculate these statistics on independent subsets of data. However, as other people noted, you will suffer efficiency losses.
Cross-validation that you've been thinking of, essentially, fights correlations between different statistics computed on the same data. There are applications where that's a dire necessity. E.g., in regression, the correlation between $\hat y_i$ and $y_i$ is given by the hat-value $h_{ii}$, the diagonal entry of the projector matrix $X(X'X)^{-1}X'$, and is $O(1)$, so you may need to suppress it when you talk about model selection or residual diagnostics (or at least control for it with degrees of freedom corrections like $n-p = n-\sum_i h_{ii}$). But there are applications where splitting the sample is a ridiculous overkill.
