9
$\begingroup$

In the comments below a post of mine, Glen_b and I were discussing how discrete distributions necessarily have dependent mean and variance.

For a normal distribution it makes sense. If I tell you $\bar{x}$, you haven't a clue what $s^2$ is, and if I tell you $s^2$, you haven't a clue what $\bar{x}$ is. (Edited to address the sample statistics, not the population parameters.)

But then for a discrete uniform distribution, doesn't the same logic apply? If I estimate the center of the endpoints, I don't know the scale, and if I estimate the scale, I don't know the center.

What's going wrong with my thinking?

EDIT

I did jbowman's simulation. Then I hit it with the probability integral transform (I think) to examine the relationship without any influence from the marginal distributions (isolation of the copula).

Data.mean <- Data.var <- rep(NA,20000)
for (i in 1:20000){     
    Data <- sample(seq(1,10,1),100,replace=T)
    Data.mean[i] <- mean(Data)
    Data.var[i] <- var(Data)    
}
par(mfrow=c(2,1))
plot(Data.mean,Data.var,main="Observations")
plot(ecdf(Data.mean)(Data.mean),ecdf(Data.var)(Data.var),main="'Copula'")

enter image description here

In the little image that appears in RStudio, the second plot looks like it has uniform coverage over the unit square, so independence. Upon zooming in, there are distinct vertical bands. I think this has to do with the discreteness and that I shouldn't read into it. I then tried it for a continuous uniform distribution on $(0,10)$.

Data.mean <- Data.var <- rep(NA,20000)
for (i in 1:20000){

    Data <- runif(100,0,10)
    Data.mean[i] <- mean(Data)
    Data.var[i] <- var(Data)

}
par(mfrow=c(2,1))
plot(Data.mean,Data.var)
plot(ecdf(Data.mean)(Data.mean),ecdf(Data.var)(Data.var))

enter image description here

This one really does look like it has points distributed uniformly across the unit square, so I remain skeptical that $\bar{x}$ and $s^2$ are independent.

$\endgroup$
  • $\begingroup$ That's an interesting approach you've taken there, I'll have to think about it. $\endgroup$ – jbowman Aug 18 '19 at 17:12
  • $\begingroup$ The dependence (necessarily) gets weaker at larger sample sizes so it's hard to see. Try smaller sample sizes, like n=5,6,7 and you'll see it more easily. $\endgroup$ – Glen_b -Reinstate Monica Aug 19 '19 at 0:09
  • $\begingroup$ @Glen_b You're right. There's a more obvious relationship when I shrink down the sample size. Even in the image I posted, there appears to be some clustering in the lower right and left corners, which is present in the plot for the smaller sample size. Two follow-ups. 1) Is the dependence necessarily getting weaker because the population parameters can be varied independent of each other? 2) It seems wrong that the statistics would have any kind of dependence, but they clearly do. What causes this? $\endgroup$ – Dave Aug 19 '19 at 0:30
  • 1
    $\begingroup$ One way to get some insight is to examine the special features of the samples that get into those 'horns" at the top of Bruce's plots. In particular note that at n=5, you get the largest possible variance by all the points being close to 0 or 1, but because there's 5 observations, you need 3 at one end and 2 at the other, so the mean must be near to 0.4 or 0.6 but not near 0.5 (since putting one point in the middle will drop the variance a bit). If you had a heavy tailed distribution, both mean and variance would be most impacted by the most extreme observation ... ctd $\endgroup$ – Glen_b -Reinstate Monica Aug 19 '19 at 0:40
  • 1
    $\begingroup$ ctd... and in that situation you get a strong correlation between $|\bar{x}-\mu|$ and $s$ (giving two big "horns" either side of the population center on a plot of sd vs mean) -- with the uniform this correlation is somewhat negative. ... With large samples you'll head toward the asymptotic behavior of $(\bar{X},s^2_X)$ which ends up being jointly normal. $\endgroup$ – Glen_b -Reinstate Monica Aug 19 '19 at 0:42
4
$\begingroup$

jbowman's Answer (+1) tells much of the story. Here is a little more.

(a) For data from a continuous uniform distribution, the sample mean and SD are uncorrelated, but not independent. The 'outlines' of the plot emphasize the dependence. Among continuous distributions, independence holds only for normal.

enter image description here

set.seed(1234)
m = 10^5; n = 5
x = runif(m*n);  DAT = matrix(x, nrow=m)
a = rowMeans(DAT)
s = apply(DAT, 1, sd)
plot(a,s, pch=".")

(b) Discrete uniform. Discreteness makes it possible to find a value $a$ of the mean and a value $s$ of the SD such that $P(\bar X = a) > 0,\, P(S = s) > 0,$ but $P(\bar X = a, X = s) = 0.$

enter image description here

set.seed(2019)
m = 20000;  n = 5;  x = sample(1:5, m*n, rep=T)
DAT = matrix(x, nrow=m)
a = rowMeans(DAT)
s = apply(DAT, 1, sd)
plot(a,s, pch=20)

(c) A rounded normal distribution is not normal. Discreteness causes dependence.

enter image description here

set.seed(1776)
m = 10^5; n = 5
x = round(rnorm(m*n, 10, 1));  DAT = matrix(x, nrow=m)
a = rowMeans(DAT);  s = apply(DAT, 1, sd)
plot(a,s, pch=20)

(d) Further to (a), using the distribution $\mathsf{Beta}(.1,.1),$ instead of $\mathsf{Beta}(1,1) \equiv \mathsf{Unif}(0,1).$ emphasizes the boundaries of the possible values of the sample mean and SD. We are 'squashing' a 5-dimensional hypercube onto 2-space. Images of some hyper-edges are clear. [Ref: The figure below is similar to Fig. 4.6 in Suess & Trumbo (2010), Intro to probability simulation and Gibbs sampling with R, Springer.]

enter image description here

set.seed(1066)
m = 10^5; n = 5
x = rbeta(m*n, .1, .1);  DAT = matrix(x, nrow=m)
a = rowMeans(DAT);  s = apply(DAT, 1, sd)
plot(a,s, pch=".")

Addendum per Comment.

enter image description here

$\endgroup$
  • $\begingroup$ Use ecdf on your last one. The scatterplot is wild! Anyway, if a uniform variable has dependence between $\bar{x}$ and $s^2$, how is it that we're getting some information about one by knowing the other, given that we can stretch the range or shift the center willy nilly and not affect the other value? If we get $\bar{x}=0$, we shouldn't know if $s^2 = 1$ or $s^2=100$, similar to how we can stretch the normal distribution without affecting the mean. $\endgroup$ – Dave Aug 18 '19 at 19:49
  • $\begingroup$ The criterion of independence is demanding. Lack of independence btw two RVs does not guarantee that it's easy to get info about one, knowing the value of the other. // In (d), not sure what ECDF of A or S would reveal. // Scatterplot in (d) shows 6 'points', images under transformation of 32 vertices of 5-d hypercube with multiplicities 1, 5, 10, 10, 5, 1 (from left to right). Multiplicities explain why 'top two' points are most distinct. $\endgroup$ – BruceET Aug 18 '19 at 20:41
  • $\begingroup$ I don't mean that it's easy to get info about one if you know the other, but if you have independence, all you can go by is the marginal distribution. Consider two standard normal variables $X$ and $Y$ with $\rho=0.9$. If you know that $x=1$, you don't know what $y$ equals, but you know that a value around $1$ is more likely than a value around $-1$. If $\rho=0$, then a value around $1$ is just as likely as a value around $-1$. $\endgroup$ – Dave Aug 18 '19 at 20:52
  • $\begingroup$ But that's for a nearly-linear relationship btw two standard normals. Mean and SD of samples are not so easy. $\endgroup$ – BruceET Aug 18 '19 at 20:59
  • 1
    $\begingroup$ @Dave you do have information about one when you know the other. For example if the sample variance is really large, you know the sample mean isn't really close to 0.5 (see the gap at the top-centre of the first plot, for example) $\endgroup$ – Glen_b -Reinstate Monica Aug 19 '19 at 0:12
2
$\begingroup$

It isn't that the mean and variance are dependent in the case of discrete distributions, it's that the sample mean and variance are dependent given the parameters of the distribution. The mean and variance themselves are fixed functions of the parameters of the distribution, and concepts such as "independence" don't apply to them. Consequently, you are asking the wrong hypothetical questions of yourself.

In the case of the discrete uniform distribution, plotting the results of 20,000 $(\bar{x}, s^2)$ pairs calculated from samples of 100 uniform $(1, 2, \dots, 10)$ variates results in:

enter image description here

which shows pretty clearly that they aren't independent; the higher values of $s^2$ are located disproportionately towards the center of the range of $\bar{x}$. (They are, however, uncorrelated; a simple symmetry argument should convince us of that.)

Of course, an example cannot prove Glen's conjecture in the post you linked to that no discrete distribution exists with independent sample means and variances!

$\endgroup$
  • $\begingroup$ That's a good catch about statistic versus parameter. I've made a pretty extensive edit. $\endgroup$ – Dave Aug 18 '19 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.