Inferring likely dates based on other related dates in incomplete data set

Question

I'm taking my first steps in data science and machine learning. I'm experimenting with a project where I have no idea even what approaches I might start with, so I'd appreciate any leads:

I have a dataset (for explanation's sake) of student graduations. The data set is complete in that it contains the entire population; all records should have a graduation date.

However, due to a record keeping failure, older records have the graduation date missing.

It has the following features:

• For graduations since 2014, we have a graduation_date
• For graduations prior to 2014, the graduation date is missing
• For all students, we have a date of birth
• For many students, graduation will be correlated to date of birth. For example, it may often that they graduate 21 years after they were born. However, some will be mature students so that they could graduate many years after the age of 21.
• The certificate IDs are more or less sequential and numeric. It can be assumed that certificate IDs close to each other therefore represent students graduating at roughly the same time
• The metaphor is somewhat flawed; assume that students can graduate on any day

My challenge is to create an approach that can infer a graduation date for all students, based on the date of birth.

The approach I have been thinking about goes something like this:

1. For all students where both dates are available, take a mode (graduation_age)
2. Group the students into bins of (say) 1000, according to the sequential certificate ID
3. Find the most common month and year of birth for the students in each bin
4. Add the mode (graduation_age) to the most common month/year for a particular bin and assign that as the graduation_date for all students in the bin

A sample in pandas might look like:

graduations = [
       # Old data with missing graduation dates
       {'certificate_id': '090029, 'birth_date': '01/01/1983', 'graduation_date': NaT},
       {'certificate_id': '090048, 'birth_date': '04/01/1983', 'graduation_date': NaT},
       ...
       # This is 'normal' students graduating roughly 21 years after
       # their birth date
       {'certificate_id': '120015, 'birth_date': '01/01/1993', 'graduation_date': 01/03/2014},
       {'certificate_id': '120019, 'birth_date': '01/04/1993', 'graduation_date': 04/03/2014},
       # However there are many exceptions, mature students or those
       # graduating early
       {'certificate_id': '120150, 'birth_date': '01/01/1966', 'graduation_date': 05/03/2014},
       {'certificate_id': '120155, 'birth_date': '01/04/1996', 'graduation_date': 06/03/2014}, 
       ]

       df = pd.DataFrame(graduations)

Any help would be appreciated, even if you are able to tell me what this sort of problem is called so that I can research further, or to let me know it is not possible with this dataset. I'm currently not even sure what the correct tags are!

Answer 1

You have described a missing data problem, and specifically one of censoring. (As a mnemonic device to keep censoring straight in my head from the similar phenomenon of truncation, I like to think of text in a report blacked-out by 'censors'. You know there was a word or sentence there, but you just don't know what it said; this is your own situation with your 'graduation dates'. By contrast, if the last 2 chapters of the report were silently omitted, then the report has been truncated. In this case, not only would you not know what was in those chapters, but you wouldn't even know if there had been any chapters. Of note, @whuber's question above was about nailing down this distinction in your data.)

In this particular missing data problem, you have what sounds like a pretty straightforward missing data mechanism: the date is missing precisely when 'graduation' occurred before 2014. If you are dealing with a time-homogenous problem lacking any important secular trends, then you can regard this fact as an advantage. In that case, you don't have a situation where data are missing for some reason that would be informative about some terribly important characteristics of the 'students'.

In missing data lingo, the specific term for what you are trying to do is to impute the missing dates. The aim of imputation is of course to permit you to retain the records with missing values, to avoid the medieval practice of so-called complete-case analysis, which involves cruelly executing the wonderful data in other fields of your data frame for the 'crime' of 'associating' with a missing date value. (I've assumed that you do in fact have numerous columns in your data which you have omitted from your example data frame; it is the existence of the valuable information in these additional columns that would justify a desire to perform such imputation.)

As far as some good reading on missing data, doing Wikipedia lookups of the various italicized terms in my answer would be a good start. The canonical reference on "Inference and missing data" is Rubin 1976. If you are of a Bayesian disposition then the fine (albeit challenging) treatment in Chapter 8 of BDA3 may be of use to you. You might instead enjoy a practical introduction to imputation through exploring software like MICE. (Sorry I'm unaware of Pythonic options in this regard, but I must suppose there are some.)

---

To address a question asked by @CliffAB in comment below, it may be helpful to contrast your chosen, imputation-based approach with other, 'fancier' approaches to censoring. The most common example of censoring in data analysis occurs in the context of survival (time-to-event) models. (See here for why this is so.) Survival models employ an estimate of the survival function, whether obtained in a parametric or non-parametric fashion, and these process models support inference without performing explicit imputation of the missing event times. You might very well attack your data with approaches like these, and never have to impute a single value!

One final point: I put 'fancier' in scare-quotes above for a reason. Suppose you made the awesomest time-to-event model ever, for your current problem. Suppose your model is so awesome, in fact, that you can't estimate is by any means short of MCMC. Your MCMC code will invariably treat the missing times as latent variables, and sure enough there will be a line in your code where you generate a pseudorandom number and use it to fill in that latent variable. Thus, you'd find yourself 'imputing' your missing data, albeit in the most highly principled and coherent way imaginable.