0
$\begingroup$

I have a dataset of flight information such as carrier, data and hour of flight, I do not have weather on this dataset yet, but I am for an external source to add to it. The dataset contains info of a year, and about 6mi different flights. I want to predict the expected delay in minutes of a given flight. The delay column contains positive and negative delays (which means that the flight departed earlier). So my question is, should I simply apply a linear regression to predict the delay? Or should I maybe use a log. reg. to predict the probability of delay and a lin. reg. to predict the expected delay and multiply the probabilities?

$\endgroup$
3
  • 1
    $\begingroup$ Please don’t cross-post. // I think you will get better answers to this interesting question here. $\endgroup$
    – Dave
    Mar 11 at 10:50
  • 1
    $\begingroup$ But you wil need to tell us some more details & context ... sample size, what covariables you do have, ... but the logistic regression idea I do not like here. $\endgroup$ Mar 11 at 11:13
  • $\begingroup$ @kjetilbhalvorsen I added some detail $\endgroup$
    – Manveru
    Mar 11 at 11:21

1 Answer 1

1
$\begingroup$

Logistic regression does not seem like a good fit, because it is intended for Yes/No (or 0/1) outcomes such as delayed vs. not delayed (see this question & discussion for whether you should model the time anyway and then classify the predictions as delayed or not, even if you were only interested in delayed vs. not delayed).

Similar to the classification vs. regression question above, I'd argue that if you have information on change in the departure time (positive or negative), that you should then model that (rather than a strictly non-negative variable, for which you set this number to 0 for on-time or departed earlier). I'd guess that you would find that doing so results in better predictions (you can and should test that using some suitable form of cross-validation - the key bit is to make sure you are evaluating all approaches you compare on the same cross-validation splits and on identical metrics - in this case perhaps something like RMSE of zero-truncated delay?). Whether linear regression (including variations of it such as using splines for some key continuous predictors) vs. other potential prediction models (e.g. tree based methods like LightGBM / XGBoost / catboost / random forest perhaps with some form of target encoding for categorical features, neural networks with embeddings for high-cardinality categorical features like airports and airlines etc., or an ensemble of multiple model types) is a good idea is another question and also worth evaluating.

There is occasionally a value in modeling two questions separately (will there be a delay + how long is the delay), but I'd suspect that in this case you'd be better off just modeling a single variable. That's because the two things are so closely related, which is less the case in some other scenarios where this has been found to be useful (e.g. in some Kaggle competitions that involve modeling amount of USD spent in response to an offer, where it can be useful to first model whether the customer will buy anything, at all, and then conditional on that the amount spent). But, again, you can look at your options and try them out using cross-validation.

$\endgroup$
2
  • $\begingroup$ Great answer. I think I ll follow your advice of modelling just as regression. But I would like to be able to justify my choice better. One idea is that if I were modelling amount of USD spent, as you mentioned, I would probably have a bimodal(?) distribution of money spend, with lots of people spending 0 and the remaning ones spending different values, while this is not the case with flight delay. Am I right? Or that is not the point? $\endgroup$
    – Manveru
    Mar 11 at 14:11
  • $\begingroup$ Yes, that's kind of what makes the cases different in my mind. Alternatively, you could say there are simply two consecutive decisions: buy anything yes/no and how much/which items to buy. That makes me consider modelling it separately (of course that's not necessarily the only sensible approach). $\endgroup$
    – Björn
    Mar 12 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.