3
$\begingroup$

The central challenge of Machine learning models is perform well on unseen data. The data is randomly split into train and test set. The test set acts as surrogate for unseen data and is used to calculate generalized error.

However, discrete choice models (such as multinomial logit model) don’t require to split the data into train and test set. They are trained on whole dataset.

Can someone please explain why don’t discrete choice models need test set. It would be great if you can also cite reference so I can read in detail later.

Thanks

Improve this question
$\endgroup$
5
  • $\begingroup$ Can you be more specific on what you mean by discrete choice models? $\endgroup$
    – Simone
    May 11 at 9:28
  • $\begingroup$ Such as binary logit model, multinomial logit model $\endgroup$
    – SiH
    May 11 at 12:44
  • 2
    $\begingroup$ IS MNL "multinomial logistic regression"? $\endgroup$
    – Dave
    May 11 at 12:44
  • $\begingroup$ Yes that is correct $\endgroup$
    – SiH
    May 11 at 12:46
  • 1
    $\begingroup$ From what I have previously discussed with some colleagues, discrete choice is more theory driven than data driven. often there is a lot of internal validation, taking subsets of the data to show validity, extrapolating data from the population using surrogate measures to make comparisons, previous studies etc. . But I also see there is a lot of pushback on this. see this sciencedirect.com/science/article/pii/S1755534520300543 $\endgroup$
    – Ralph Winters
    May 11 at 13:24

1 Answer 1

Reset to default
3
$\begingroup$

Multinomial logistic regression can and often does consider out-of-sample performance. For instance, LeCun (1998) applied multinomial logistic regression to the pixels of the MNIST handwritten digits, achieving an accuracy of $88\%$ on the test set.

As is discussed on the blog of our Demetri Pananos, biostatistics has a different take on model validation than out-of-sample performance, preferring to use bootstrap-based approaches, but that is not unique to models of discrete outcomes ("classification"), and there is an interesting debate about out-of-sample performance vs bootstrap (linked in the blog).

LeCun, Yann, et al. "Gradient-based learning applied to document recognition." Proceedings of the IEEE 86.11 (1998): 2278-2324.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you very much. Can you please provide some argument in favor of discrete choice models justifying them for not using test set. $\endgroup$
    – SiH
    May 11 at 13:07
  • $\begingroup$ @SiH I don't see any arguments in favor of in-sample assessments for classification models that wouldn't apply to regression models. The big advantage is the increase in sample size. // The bootstrap-based approach in my link is worth a read. $\endgroup$
    – Dave
    May 11 at 18:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.