# Concept of a random linear model

I want to compare few methods of variable selection. I want to do it using simulations. I'm aware of the fact that it won't be an ultimate answer to question 'Which method is the best one?', but I'm looking just form some hint. To do such simulations I need a method to draw a 'random linear model'. Is there any well accepted algorithm of drawing 'random linear model'? By well accepted I mean method that was used for example in some scientific paper.

I was thinking about following simple approach:
1) Choose $n$ and $k$, which denotes number of observations and number of variables.
2) Generate random matrix $X$ by drawing each element using uniform distribution $(0,1)$.
3) Generate parameters using uniform distribution $(0,1)$.
4) Generate residuals using Normal Distribution $(0,\sigma^2)$ for some fixed and arbitrary chosen $\sigma^2$
5) Calculate $Y=X\beta + \epsilon$

-

One problem with the approach you outline is that the regressors $x_i$ will (on average) be uncorrelated, and one situation in which variable selection methods have difficulty is highly correlated regressors.

I'm not sure the concept of a 'random' linear model is very useful here, as you have to decide on a probability distribution over your model space, which seems arbitrary. I'd rather think of it as an experiment, and apply the principles of good experimental design.

Postscript: Here's one reference but i'm sure there are others:
Andrea Burton, Douglas G. Altman, Patrick Royston, and Roger L. Holder. The design of simulation studies in medical statistics. Statistics in Medicine 25(24):4279-4292, 2006. DOI:10.1002/sim.2673

Hakan Demirtas. Statistics in Medicine 26(20):3818-3821, 2007. DOI:10.1002/sim.2876

Just found a commentary on a similar topic:
G. Maldonado and S. Greenland. The importance of critically interpreting simulation studies. Epidemiology 8 (4):453-456, 1997. http://www.jstor.org/stable/3702591

-
I know that they will be uncorrelated, but still it would be useful to know what is a good approach to such experiments. –  Tomek Tarczynski Feb 2 '11 at 15:27
I've just added a ref to my answer above. –  onestop Feb 2 '11 at 16:17
+1 Good advice and good references. It's difficult to think of what use a "random" model would have: no method is universal, so one needs to test it on data that are characteristic of the intended areas of application. –  whuber Feb 2 '11 at 17:28

To address @onestop's objection to non-correlated regressors, you could do the following:

1. Choose $n, k, l$, where $l$ is the number of latent factors.
2. Choose $\sigma_i$, the amount of 'idiosyncratic' volatility in the regressors.
3. Draw a $k \times l$ matrix, $F$, of exposures, uniformly on $(0,1)$. (you may want to normalize to sum 1 across rows of $F$.)
4. Draw a $n \times l$ matrix, $W$, of latent regressors as standard normals.
5. Let $X = W F^\top + \sigma_i E$ be the regressors, where $E$ is an $n\times k$ matrix drawn from a standard normal.
6. Proceed as before: draw $k$ vector $\beta$ uniformly on $(0,1)$.
7. draw $n$ vector $\epsilon$ as a normal with variance $\sigma^2$.
8. Let $y = X\beta + \epsilon$.
-
Good to know, it also might be helpful. Thanks! –  Tomek Tarczynski Feb 3 '11 at 13:11