1
$\begingroup$

Say I want to compare three methods in a simulation study. Let’s say I want to compare the lasso, the group lasso, and a neural network at selecting relevant variables.

In terms of validation metrics, I could use accuracy, FDR, etc. However, how do I pick the parameters for each approach to allow for fair comparison using these metrics?

The two lasso approaches share many parameters, so those we can simply replicate. On the other hand, the NN approach would have completely different parameters and would likely be the most expensive approach, so would I need to pick parameters that mean the approach takes roughly the same amount of computational time to run as the other two?

Is there a general rule I should be following here? The only one I can think of that makes sense is to ensure the computational times are roughly the same, but this is difficult to ensure across a full simulation study.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Welcome to Cross Validated! What is a “fair comparison” in your mind? I would expect you to be interested in which approach is the best at picking the relevant variables, but you seem to have concerns about computing time, too. What do you aim to assess? $\endgroup$
    – Dave
    Jul 21, 2022 at 15:49
  • $\begingroup$ @Dave I am thinking of a comparison that a practitioner could use. If they want an accurate method but also one that doesn’t take forever to run. I’m thinking about this because each method has a parameter in relation to the fitting algorithm. So each method can be made more accurate by running the algorithm longer and/or having a smaller convergence threshold. $\endgroup$
    – Tom
    Jul 21, 2022 at 18:09

1 Answer 1

Reset to default
1
$\begingroup$

Unless you think that computational time is a validation metric itself, you shouldn't be worrying about that initially. You might consider that in an evaluation of the tradeoffs among modeling approaches later. If a neural net works 10% better than group lasso but takes 100 times as long, that's something one needs to consider in terms of costs of imprecision versus costs of computation and waiting for results.

What's most important is to simulate data that represent a scenario that will be related to one of practical interest. The "best" method typically depends on the nature of the data. Then assess each of the approaches while you apply its own individual best practices for model-parameter selection.

Use a "validation metric" appropriate to the nature of the outcome variable. For continuous outcomes and constant expected error variance, mean-square error is the standard. Accuracy is not a good choice even for what are considered "classification" models; you typically want to evaluate the validity of probabilities of class assignment, while class assignments from models often involve an arbitrary and hidden choice of a probability cutoff.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Each model has a parameter in relation to the tuning algorithm. So generally, running it longer = better results. I’m trying to ensure my simulation study has a realistic overall run time, so have to make sacrifices across the board. But it may be that for 100 iterations, a NN takes 10x longer than for 100 iterations of the group lasso fitting process. So how do I decide on these fitting parameters? $\endgroup$
    – Tom
    Jul 21, 2022 at 18:11
  • $\begingroup$ @Tom training-set performance might seem to improve with longer run times, but test-set or cross-validated performance usually doesn't. See for example Figure 2.9 of ISLR, 2nd edition, where you might consider model "flexibility" as a proxy for run time. For LASSO and group LASSO, the tuning (penalty) parameter is typically chosen by cross-validation on the data set--that's the type of thing I meant by best practices. Increasing run times wouldn't appreciably help with that. Figure 10.11 illustrates this for a comparison of LASSO against a neural net. $\endgroup$
    – EdM
    Jul 21, 2022 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.