I now ran into two variables that give me problems. I know that I want to try to include them in the final model, but I'm unsure how the values should be treated in order to reflect the true nature of the phenomena they are reflecting.

Case 1

The variable depicts the age/duration of a certain contract type. If the customer subscribes to this specific type, the variable shows the age of this contract since it start. However, if the customer doesn't have this type of contract, the value is zero. If I use this variable straight out of the box as a integer, you see the problem: the value "0" is quite different from any number of days (although it is in a way a correct answer). How should/could this variable be preprocessed? Durations go from days to several years.

Case 2

In my modeling I use a dataset that goes back some 3 years. I then based on this dataset calculate if and how long a go, certain changes happened. If there has been a change, I can include the change and the time elapsed since this change. But what is the best way to include these variables? Missing values indicate no change (although a cange might well have occured, but outside the collected data), while a integer indicates n days since a change. Both cases carry valuable information, but the data is presented in slightly different form.


The easiest way to think about your Case 1 problem is that you have a predictor variable where you are anticipating a highly non-linear relationship between it and the dependent variable. There are lots of solutions to this problem.

The simplest is to add a binary variable to your model which takes a value of 1 when your variable is 0 and a value of 0 otherwise.

A more complicated solution which also will address other non-linearity issues is to use a spline. I am particularly partial to thin plated regression splines, which are available in a library in R (Wood, S.N. (2003), "Thin plate regression splines," Journal of the Royal Statistical Society, Series B (Methodological), 65 (1), 95-114.). However, I have not checked to see if there is anything newer or better than this in recent years, so it may have been superseded.

Your case 2 problem is one of censoring and is a lot more complicated as it is essentially a question of estimating a model which takes into account that you have some lack of certainty about the truth for some of your data. There is unlikely to be a simple and neat trick to pre-processing the data which addresses this. You could perhaps try the same idea as I have proposed for Case 1, and that may be OK if your goal is just one of prediction. However, if you need to interpret the coefficients then you may need to have a more rigorous solution. If you do a google for "censored independent variables" you will find more information.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.