How best to deal with missing event data

Question

I have a data set which contains values like "last_foo" containing the number of days since the last time foo occurred. Naturally this feature contains NaNs for examples that have never foo'd. What's the best way to address this so I can use these features with scikit-learn models that don't allow missing data? I can't replace with 0 since that would mean foo happened 0 days ago, which is valid data here.

I thought I might use KBinsDiscretizer to bin the data into one-hot features and for the examples with no foo events they wouldn't have any of the bins set to 1, but KBinsDiscretizer doesn't work on nulls. My next idea is to replace NaNs with the max value or possibly max + some amount, but that seems wrong. Same for mean replacement, which is certainly not right - the fact that foo never happened is significant to the system I'm trying to analyze.

Are there other approaches I should consider? Thank you!

Answer 1

As @whuber mentioned, this is censored data. The lifelines python library is a very good one for survival analysis. It has excellent documentation that is a good place to start learning about the topic.

Here are some excerpts from the docs:

The individuals in a population who have not been subject to the death event are labeled as right-censored, i.e., we did not (or can not) view the rest of their life history due to some external circumstances. All the information we have on these individuals are their current lifetime durations (which is naturally less than their actual lifetimes).

A common mistake data analysts make is choosing to ignore the right-censored individuals. We will see why this is a mistake next.

Furthermore, if we instead simply took the mean of all lifespans, including the current lifespans of right-censored instances, we would still be underestimating the true average lifespan.

Another package option is scikit-survival.

One final note: consider whether the event foo can only happen once to each population unit (like death), or if it can recur (like pregnancy). If the latter is true, you might have to look into methods for recurrent event analysis.