6
$\begingroup$

I am confused with this logical thinking: enter image description here

The quote is from the book “The art of ML” by Matloff

He is working with the dataset https://www.kaggle.com/datasets/joniarroba/noshowappointments

I agree with ID columns, but… Why would you not include neighborhood? It seems a good feature, (neighborhoods with bad public transport might get their transport canceled…) Appointment day and scheduled day also seem important…if the appointment was scheduled many months in advance, the patient might forget… Also, if the day is a Friday, maybe people are more forgetful on Fridays… Why is the author discarding useful data?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ The claim that the data are useful isn't established. All of the hypotheses you've outlined are things that you can test, but you would have to actually carry out that analysis to establish whether or not the data is "useful." The author's justification is right there in the quotation: "using the same reasoning..." suggests that the author believes this data is too sparse. Your analysis could establish whether or not that's true, but the author directly gives their justification. $\endgroup$
    – Sycorax ♦
    Commented Jan 4 at 14:25
  • $\begingroup$ So using the author's justification, extracting day of week or month from 'AppointmentDay' would be fine (on the assumption enough of each) $\endgroup$
    – seanv507
    Commented Jan 4 at 16:55

1 Answer 1

Reset to default
2
$\begingroup$

You're allowed to disagree with what the author claims. For instance, at least out of context, there is an argument to be made that even better than dropping these variables would be to use a regularized regression that will penalize the model for having many variables (relatively high model complexity) while allowing for the possibility that these variables do wind up being useful. If you don't want to regularize (for whatever reason), however, you might choose to exclude some variables to keep the model from being overly complex, especially if you have limited data.

To answer explicitly...

Why is the author discarding useful data?

...the author seems to believe these to be among the less important variables in determining the outcome and chooses to exclude them from the regression, favoring the possible loss of information in those variables over model complexity that risks overfitting when those variables are included.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.