Concept of a random linear model

Question

I want to compare few methods of variable selection. I want to do it using simulations. I'm aware of the fact that it won't be an ultimate answer to question 'Which method is the best one?', but I'm looking just form some hint. To do such simulations I need a method to draw a 'random linear model'. Is there any well accepted algorithm of drawing 'random linear model'? By well accepted I mean method that was used for example in some scientific paper.

I was thinking about following simple approach:
1) Choose $n$ and $k$, which denotes number of observations and number of variables.
2) Generate random matrix $X$ by drawing each element using uniform distribution $(0,1)$.
3) Generate parameters using uniform distribution $(0,1)$.
4) Generate residuals using Normal Distribution $(0,\sigma^2)$ for some fixed and arbitrary chosen $\sigma^2$
5) Calculate $Y=X\beta + \epsilon$

Answer 1

One problem with the approach you outline is that the regressors $x_i$ will (on average) be uncorrelated, and one situation in which variable selection methods have difficulty is highly correlated regressors.

I'm not sure the concept of a 'random' linear model is very useful here, as you have to decide on a probability distribution over your model space, which seems arbitrary. I'd rather think of it as an experiment, and apply the principles of good experimental design.

Postscript: Here's one reference but i'm sure there are others:
Andrea Burton, Douglas G. Altman, Patrick Royston, and Roger L. Holder. The design of simulation studies in medical statistics. Statistics in Medicine 25(24):4279-4292, 2006. DOI:10.1002/sim.2673

See also this related letter:
Hakan Demirtas. Statistics in Medicine 26(20):3818-3821, 2007. DOI:10.1002/sim.2876

Just found a commentary on a similar topic:
G. Maldonado and S. Greenland. The importance of critically interpreting simulation studies. Epidemiology 8 (4):453-456, 1997. http://www.jstor.org/stable/3702591

Answer 2

To address @onestop's objection to non-correlated regressors, you could do the following:

1. Choose $n, k, l$, where $l$ is the number of latent factors.
2. Choose $\sigma_i$, the amount of 'idiosyncratic' volatility in the regressors.
3. Draw a $k \times l$ matrix, $F$, of exposures, uniformly on $(0,1)$. (you may want to normalize to sum 1 across rows of $F$.)
4. Draw a $n \times l$ matrix, $W$, of latent regressors as standard normals.
5. Let $X = W F^\top + \sigma_i E$ be the regressors, where $E$ is an $n\times k$ matrix drawn from a standard normal.
6. Proceed as before: draw $k$ vector $\beta$ uniformly on $(0,1)$.
7. draw $n$ vector $\epsilon$ as a normal with variance $\sigma^2$.
8. Let $y = X\beta + \epsilon$.