5
$\begingroup$

I recently had a puzzling interview question and I am wondering whether anybody can tell me the intended answer.

The question shows train and test error for three models plotted against the number of iterations/epochs and asks you to rank the models in order "from best to worst".

All models have the same minimum test error but different error curves on the training set.

It is a multiple choice question and all six of the orders are given as possible answers.

I didn't know what the intention was (or what "best" means) so I ranked the models by how many epochs they took to achieve minimum test set error. But this is just a guess.

enter image description here

What is the intended answer to the question?

$\endgroup$
4
  • $\begingroup$ Interesting question. I would've also chosen 2-1-3 based on the fact that model 2 has the smallest vertical difference between train and val error at its chosen epoch, followed by model 1 and then model 3. However, I am not sure if my reasoning is correct. $\endgroup$
    – mhdadk
    Feb 22, 2021 at 3:18
  • 1
    $\begingroup$ Update: I found out that 2-1-3 is correct, but I don't yet know why. I will ask at the interview if I get a chance. $\endgroup$
    – Flounderer
    Feb 24, 2021 at 1:49
  • $\begingroup$ Thanks for the update! Let me know if you find out the reasoning behind this answer. $\endgroup$
    – mhdadk
    Feb 24, 2021 at 9:19
  • 1
    $\begingroup$ Final update: I didn't get the job due to having insufficient evidence of leadership, so I guess I'll never know! $\endgroup$
    – Flounderer
    Apr 13, 2021 at 3:48

1 Answer 1

1
$\begingroup$

I would agree that 2-1-3 seems like the best model, but I think there a couple other factors that could be considered which provide additional support for that answer.

  1. The fact that the training curve is less convex suggests that the model is more robust to overfitting.

  2. The train curve stays closer to the test curve than for the other models, suggesting that the model is learning the "right" information from the training set.

  3. In addition to reaching its minimum quickest, model 2's loss is lower already after the first epoch. Depending on the setup of this problem, this could possibly suggest the model is better able to extract useful information.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.