How to perform a fair simulation study? Say I want to compare three methods in a simulation study. Let’s say I want to compare the lasso, the group lasso, and a neural network at selecting relevant variables.
In terms of validation metrics, I could use accuracy, FDR, etc. However, how do I pick the parameters for each approach to allow for fair comparison using these metrics?
The two lasso approaches share many parameters, so those we can simply replicate. On the other hand, the NN approach would have completely different parameters and would likely be the most expensive approach, so would I need to pick parameters that mean the approach takes roughly the same amount of computational time to run as the other two?
Is there a general rule I should be following here? The only one I can think of that makes sense is to ensure the computational times are roughly the same, but this is difficult to ensure across a full simulation study.
 A: Unless you think that computational time is a validation metric itself, you shouldn't be worrying about that initially. You might consider that in an evaluation of the tradeoffs among modeling approaches later. If a neural net works 10% better than group lasso but takes 100 times as long, that's something one needs to consider in terms of costs of imprecision versus costs of computation and waiting for results.
What's most important is to simulate data that represent a scenario that will be related to one of practical interest. The "best" method typically depends on the nature of the data. Then assess each of the approaches while you apply its own individual best practices for model-parameter selection.
Use a "validation metric" appropriate to the nature of the outcome variable. For continuous outcomes and constant expected error variance, mean-square error is the standard. Accuracy is not a good choice even for what are considered "classification" models; you typically want to evaluate the validity of probabilities of class assignment, while class assignments from models often involve an arbitrary and hidden choice of a probability cutoff.
