I'm taking my first steps in data science and machine learning. I'm experimenting with a project where I have no idea even what approaches I might start with, so I'd appreciate any leads:

I have a dataset (for explanation's sake) of student graduations. The data set is complete in that it contains the entire population; all records should have a graduation date.

However, due to a record keeping failure, older records have the graduation date missing.

It has the following features:

  • For graduations since 2014, we have a graduation_date
  • For graduations prior to 2014, the graduation date is missing
  • For all students, we have a date of birth
  • For many students, graduation will be correlated to date of birth. For example, it may often that they graduate 21 years after they were born. However, some will be mature students so that they could graduate many years after the age of 21.
  • The certificate IDs are more or less sequential and numeric. It can be assumed that certificate IDs close to each other therefore represent students graduating at roughly the same time
  • The metaphor is somewhat flawed; assume that students can graduate on any day

My challenge is to create an approach that can infer a graduation date for all students, based on the date of birth.

The approach I have been thinking about goes something like this:

  1. For all students where both dates are available, take a mode (graduation_age)
  2. Group the students into bins of (say) 1000, according to the sequential certificate ID
  3. Find the most common month and year of birth for the students in each bin
  4. Add the mode (graduation_age) to the most common month/year for a particular bin and assign that as the graduation_date for all students in the bin

A sample in pandas might look like:

graduations = [
       # Old data with missing graduation dates
       {'certificate_id': '090029, 'birth_date': '01/01/1983', 'graduation_date': NaT},
       {'certificate_id': '090048, 'birth_date': '04/01/1983', 'graduation_date': NaT},
       # This is 'normal' students graduating roughly 21 years after
       # their birth date
       {'certificate_id': '120015, 'birth_date': '01/01/1993', 'graduation_date': 01/03/2014},
       {'certificate_id': '120019, 'birth_date': '01/04/1993', 'graduation_date': 04/03/2014},
       # However there are many exceptions, mature students or those
       # graduating early
       {'certificate_id': '120150, 'birth_date': '01/01/1966', 'graduation_date': 05/03/2014},
       {'certificate_id': '120155, 'birth_date': '01/04/1996', 'graduation_date': 06/03/2014}, 

       df = pd.DataFrame(graduations)

Any help would be appreciated, even if you are able to tell me what this sort of problem is called so that I can research further, or to let me know it is not possible with this dataset. I'm currently not even sure what the correct tags are!

  • 4
    $\begingroup$ You have some impressive opportunities for creating profound but subtle biases. Perhaps the most important one to recognize concerns the bias in your subsample of known graduation dates. This is not representative, because there may still be many (older) students who haven't yet graduated--and therefore will tend to take longer than most. It's akin to the well-known cemetery problem: when you glean birth and death dates from gravestones, you will conclude that recently lifespans have been growing dramatically shorter! $\endgroup$ – whuber Jan 1 '16 at 14:59
  • $\begingroup$ Thank you for your comment. I think that I may have overstretched an analogy in an effort to make things simpler. In this case, it can be assumed that all records in the dataset should have a graduation date. That is, they are historical records - the data is incomplete due to an error in record keeping rather than sampling. The record set is a complete population, just with missing data in one column. $\endgroup$ – somewhatoff Jan 2 '16 at 18:20
  • 1
    $\begingroup$ That information is of such fundamental importance that you should edit your post to include it. $\endgroup$ – whuber Jan 2 '16 at 18:41
  • 1
    $\begingroup$ Thank you for you comment @DavidC.Norris; generally I think it is correct in pandas to use NaN for NULL. See: pandas.pydata.org/pandas-docs/stable/missing_data.html. However, reading that again I discovered that technically I should have NaT (not a time) - I have edited this. $\endgroup$ – somewhatoff Jan 6 '16 at 10:37
  • 2
    $\begingroup$ People have subtly hinted and I think I would actually state: If you can rethink your analysis to treat this data as censored rather than by attempting to impute dates, I really think you ought to do that. You do not give enough details to know whether this is a reasonable suggestion. But if what you are using the date for is to compute how long they took to graduate, then that is a censoring problem can can be treated in many other ways. $\endgroup$ – StatNoodle Jan 6 '16 at 19:56

You have described a missing data problem, and specifically one of censoring. (As a mnemonic device to keep censoring straight in my head from the similar phenomenon of truncation, I like to think of text in a report blacked-out by 'censors'. You know there was a word or sentence there, but you just don't know what it said; this is your own situation with your 'graduation dates'. By contrast, if the last 2 chapters of the report were silently omitted, then the report has been truncated. In this case, not only would you not know what was in those chapters, but you wouldn't even know if there had been any chapters. Of note, @whuber's question above was about nailing down this distinction in your data.)

In this particular missing data problem, you have what sounds like a pretty straightforward missing data mechanism: the date is missing precisely when 'graduation' occurred before 2014. If you are dealing with a time-homogenous problem lacking any important secular trends, then you can regard this fact as an advantage. In that case, you don't have a situation where data are missing for some reason that would be informative about some terribly important characteristics of the 'students'.

In missing data lingo, the specific term for what you are trying to do is to impute the missing dates. The aim of imputation is of course to permit you to retain the records with missing values, to avoid the medieval practice of so-called complete-case analysis, which involves cruelly executing the wonderful data in other fields of your data frame for the 'crime' of 'associating' with a missing date value. (I've assumed that you do in fact have numerous columns in your data which you have omitted from your example data frame; it is the existence of the valuable information in these additional columns that would justify a desire to perform such imputation.)

As far as some good reading on missing data, doing Wikipedia lookups of the various italicized terms in my answer would be a good start. The canonical reference on "Inference and missing data" is Rubin 1976. If you are of a Bayesian disposition then the fine (albeit challenging) treatment in Chapter 8 of BDA3 may be of use to you. You might instead enjoy a practical introduction to imputation through exploring software like MICE. (Sorry I'm unaware of Pythonic options in this regard, but I must suppose there are some.)

To address a question asked by @CliffAB in comment below, it may be helpful to contrast your chosen, imputation-based approach with other, 'fancier' approaches to censoring. The most common example of censoring in data analysis occurs in the context of survival (time-to-event) models. (See here for why this is so.) Survival models employ an estimate of the survival function, whether obtained in a parametric or non-parametric fashion, and these process models support inference without performing explicit imputation of the missing event times. You might very well attack your data with approaches like these, and never have to impute a single value!

One final point: I put 'fancier' in scare-quotes above for a reason. Suppose you made the awesomest time-to-event model ever, for your current problem. Suppose your model is so awesome, in fact, that you can't estimate is by any means short of MCMC. Your MCMC code will invariably treat the missing times as latent variables, and sure enough there will be a line in your code where you generate a pseudorandom number and use it to fill in that latent variable. Thus, you'd find yourself 'imputing' your missing data, albeit in the most highly principled and coherent way imaginable.

  • 1
    $\begingroup$ A couple of notes: while I'm pretty sure I know what you mean about censoring vs truncation, I think it's really confusing to include an example that sounds like censorship when describing censoring. Secondly, censoring is a very special type of missing data: it's partially informative, unlike NA which completely uninformative. It's unclear to me at the moment how MICE hands censored data, but I believe you can. Are you aware of how MICE handles it? $\endgroup$ – Cliff AB Jan 9 '16 at 7:00
  • 1
    $\begingroup$ @CliffAB, thank you for your comment + question. I have clarified that all that talk about 'censorship' was purely my humble offering of a mnemonic device. The question of proper coding idioms for data (the NA issue) should be distinguished from issues of modeling. Both truncated and censored data would be coded as NA in R (or . in SAS). But how the modeler treats such 'entries' depends on the nature/mechanism of 'missingness'. $\endgroup$ – David C. Norris Jan 10 '16 at 12:46
  • $\begingroup$ @CliffAB, when I use imputation myself, it is typically via Frank Harrell's Hmisc package (via rms). My citation of MICE was an attempt to point OP to a package with a special focus on OP's problem, but it is my understanding that rms uses MICE-like algorithms. $\endgroup$ – David C. Norris Jan 10 '16 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.