Say I want to compare three methods in a simulation study. Let’s say I want to compare the lasso, the group lasso, and a neural network at selecting relevant variables.
In terms of validation metrics, I could use accuracy, FDR, etc. However, how do I pick the parameters for each approach to allow for fair comparison using these metrics?
The two lasso approaches share many parameters, so those we can simply replicate. On the other hand, the NN approach would have completely different parameters and would likely be the most expensive approach, so would I need to pick parameters that mean the approach takes roughly the same amount of computational time to run as the other two?
Is there a general rule I should be following here? The only one I can think of that makes sense is to ensure the computational times are roughly the same, but this is difficult to ensure across a full simulation study.