1
$\begingroup$

I am trying to implement a Simulation from the book "Elements of Statistical Learning" by Hastie et al.

My Problem is that I don't understand how to generate the pseudodata as they did. The book says

For each of N = 100 Samples, we generated p standard Gaussian features X with pairwise correlation 0.2. The outcome Y was generated according to a linear model* $$Y = \sum_{j=1}^p X_j \beta_j + \sigma \epsilon,$$ *where $\epsilon$ was generated from a Standard Gaussian Distribution. For each dataset, the set of coefficients $\beta_j$ were also generated from a Standard Gaussian Distribution. We investigated p = 20, 100 and 1000. The standard deviation $ \sigma $ was chosen in each case so that the signal-to-noise-ratio $Var[E(Y|X)]/ \sigma ^2 $equaled 2.

So, what I managed to generate so far are the Xs, the $\epsilon$ and the $\beta$s.

I don't get how I'm meant to generate Y without knowing $\sigma$ and according to the description of $\sigma$, I need Y to compute it.

Can someone please help me? What am I not understanding here?? Thanks and best regards!

$\endgroup$
  • $\begingroup$ Math mode is accessed by placing $\TeX$ between dollar sign delimiters or double dollar signs to set off an equation on its own line. I edited part of your post to illustrate. $\endgroup$ – whuber Dec 9 '17 at 18:40
  • $\begingroup$ @whuber Thanks for the Information, looks much better now :) $\endgroup$ – machinelearner Dec 9 '17 at 20:11
1
$\begingroup$

The standard deviation sigma was chosen in each case so that the signal-to-noise-ratio $Var(E[Y|X]) / \sigma^2$ equaled 2.

Because $\epsilon$ has mean 0, we know that:

$$E[Y \mid X] = \sum_{j=1}^p X_j \beta_j = \beta^T X$$

So, using the $X$ and $\beta$ you generated, calculate the variance of $E[Y \mid X]$ and divide it by 2 to obtain $\sigma^2$.

Clarification: this should be done by treating $X$ as a random variable, not by working with samples. $E[Y \mid X]$ is a linear combination of Gaussian random variables so, as described here and by whuber in the comments below:

$$Var(E[Y \mid X]) = \beta^T C \beta$$

where $C$ is the covariance matrix of $X$

$\endgroup$
  • $\begingroup$ Thank you for answering! I'm still not entirely sure what you mean: "calculate $E(Y|X)$ for each value of $X$" - do you rather mean to compute $E(Y_i|X)= \sum_{j=1}^{p} X_{ij}b_j$ (for i = 1,...,100) or do you mean something like $E(Y|X_?)$ ? $\endgroup$ – machinelearner Dec 9 '17 at 19:50
  • 1
    $\begingroup$ The expression "$\operatorname{Var}(E[Y\mid X])$" is a variance taken with respect to the distribution of the random variable $X$. Thus, a prescription to use only a particular realization of $X$ would be at best an approximation to what was intended. Instead, once the $\beta_j$ have been generated, you need to work out this variance mathematically and use that value to determine $\sigma^2$. That's straightforward to do using the rules for computing variances and the fact that you know the variance-covariance matrix of $X$. $\endgroup$ – whuber Dec 9 '17 at 21:20
  • $\begingroup$ @whuber so this is what I get mathematically and then try to implement in R: beta = data.frame(rnorm(20)) for (i in 1:20){ for (j in 1:20){ if (i != j){ a=sum(beta[i,1]^2)+0.2*sum(beta[i,1]*beta[j,1]);}}} Is that any good? Your help is much appreciated, thank you! $\endgroup$ – machinelearner Dec 9 '17 at 22:50
  • $\begingroup$ No amount of code is going to make up for the difference between $X$ as a random variable, which is the interpretation demanded by the notation in the quotation, and $X$ as a realization of random values, which is what you are doing. $\endgroup$ – whuber Dec 9 '17 at 22:51
  • $\begingroup$ @whuber I didn't use any realizations of $X$ this time, only of $\beta$. I used 0.2 for the covariances of the $X$-pairs and 1 for the Variance of each $X$. Please correct me if that's not what you intended to explain earlier? $\endgroup$ – machinelearner Dec 9 '17 at 22:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.