Many times in data that stems from transaction systems (OLTP), a categorical type of value is included in a numeric field. For example, if you have the date of some event occurring, if the event has never occurred, there will be a placeholder date (e.g. 1900-01-01).

In my current analysis, I have a days since last event, where the majority of records have 999 to indicate "never occurred".

I have been unable to find any research/information on the best way to handle this type of value. Treating these values like null values and using some sort of imputation makes no sense at all, since the values actually have a meaning; using other records to estimate what these fields should be would totally remove the information to be gained from these values.

However, I am afraid that using any distance based ML algorithm would be thrown off by these usually extreme values.

My current thought is to create additional dummy variables for y/n if the even occurred or not, but do not know what to do with these extreme values in the original field. Any suggestions or research I could read would be appreciated.

  • $\begingroup$ I'm assuming you want to analyze how long something takes. Have you read about survival or time-to-event analysis? One way to think about these data points is that they are censored, in which case if you can incorporate their information. $\endgroup$ – robin.datadrivers Nov 14 '16 at 16:45
  • $\begingroup$ @robin.datadrivers, hi, the current analysis is a classification problem where I would like to determine which person will accept an offer. The days since last event field is actually "days since last contact by company to customer" which is one of many independent variables. So, this is particular problem not a survival analysis. My main question is how to deal with numerical independent variables which have an arbitrary value indicating a non-event (in the case of "customer never contacted by company"). $\endgroup$ – ivan7707 Nov 14 '16 at 16:55
  • $\begingroup$ I think your approach to have a dummy variable of yes/no sounds reasonable, and then code the 999s as 0s. Effectively, this makes the interpretation of the variable "Time since last contact" an interaction term, interpreted as the effect of Time since last contact, given that a contact was made. $\endgroup$ – robin.datadrivers Nov 14 '16 at 17:10
  • $\begingroup$ @robin.datadrivers, thank you. I will also read more about interaction terms. $\endgroup$ – ivan7707 Nov 14 '16 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.