# Generating pseudodata as in “Elements of Statistical Learning”

I am trying to implement a Simulation from the book "Elements of Statistical Learning" by Hastie et al.

My Problem is that I don't understand how to generate the pseudodata as they did. The book says

For each of N = 100 Samples, we generated p standard Gaussian features X with pairwise correlation 0.2. The outcome Y was generated according to a linear model* $$Y = \sum_{j=1}^p X_j \beta_j + \sigma \epsilon,$$ *where $\epsilon$ was generated from a Standard Gaussian Distribution. For each dataset, the set of coefficients $\beta_j$ were also generated from a Standard Gaussian Distribution. We investigated p = 20, 100 and 1000. The standard deviation $\sigma$ was chosen in each case so that the signal-to-noise-ratio $Var[E(Y|X)]/ \sigma ^2$equaled 2.

So, what I managed to generate so far are the Xs, the $\epsilon$ and the $\beta$s.

I don't get how I'm meant to generate Y without knowing $\sigma$ and according to the description of $\sigma$, I need Y to compute it.

• Math mode is accessed by placing $\TeX$ between dollar sign delimiters or double dollar signs to set off an equation on its own line. I edited part of your post to illustrate. – whuber Dec 9 '17 at 18:40
• @whuber Thanks for the Information, looks much better now :) – machinelearner Dec 9 '17 at 20:11

The standard deviation sigma was chosen in each case so that the signal-to-noise-ratio $Var(E[Y|X]) / \sigma^2$ equaled 2.

Because $\epsilon$ has mean 0, we know that:

$$E[Y \mid X] = \sum_{j=1}^p X_j \beta_j = \beta^T X$$

So, using the $X$ and $\beta$ you generated, calculate the variance of $E[Y \mid X]$ and divide it by 2 to obtain $\sigma^2$.

Clarification: this should be done by treating $X$ as a random variable, not by working with samples. $E[Y \mid X]$ is a linear combination of Gaussian random variables so, as described here and by whuber in the comments below:

$$Var(E[Y \mid X]) = \beta^T C \beta$$

where $C$ is the covariance matrix of $X$

• Thank you for answering! I'm still not entirely sure what you mean: "calculate $E(Y|X)$ for each value of $X$" - do you rather mean to compute $E(Y_i|X)= \sum_{j=1}^{p} X_{ij}b_j$ (for i = 1,...,100) or do you mean something like $E(Y|X_?)$ ? – machinelearner Dec 9 '17 at 19:50
• The expression "$\operatorname{Var}(E[Y\mid X])$" is a variance taken with respect to the distribution of the random variable $X$. Thus, a prescription to use only a particular realization of $X$ would be at best an approximation to what was intended. Instead, once the $\beta_j$ have been generated, you need to work out this variance mathematically and use that value to determine $\sigma^2$. That's straightforward to do using the rules for computing variances and the fact that you know the variance-covariance matrix of $X$. – whuber Dec 9 '17 at 21:20
• @whuber so this is what I get mathematically and then try to implement in R: beta = data.frame(rnorm(20)) for (i in 1:20){ for (j in 1:20){ if (i != j){ a=sum(beta[i,1]^2)+0.2*sum(beta[i,1]*beta[j,1]);}}} Is that any good? Your help is much appreciated, thank you! – machinelearner Dec 9 '17 at 22:50
• No amount of code is going to make up for the difference between $X$ as a random variable, which is the interpretation demanded by the notation in the quotation, and $X$ as a realization of random values, which is what you are doing. – whuber Dec 9 '17 at 22:51
• @whuber I didn't use any realizations of $X$ this time, only of $\beta$. I used 0.2 for the covariances of the $X$-pairs and 1 for the Variance of each $X$. Please correct me if that's not what you intended to explain earlier? – machinelearner Dec 9 '17 at 22:54