10
$\begingroup$

I'm in a bit of a quandry with a logistic model I'm working on. As one of the explanatory variables, I want to include "Days since last visit" (or some transformation of it), however about 20% of the population I'm predicting for has never made a visit so their value for this would be Null.

Options I'm aware of:

  • Setting Nulls to -/+ ∞. I'd prefer to avoid doing this since it's essentially writing off the prediction for 20% of my population as 0 or 1

  • Adding a Categorical variable for "Is Null" and regressing with that as well. This seems like the most intuitive to me, but I'm still not sure on how I'd transform the model to capture the continuous element (i.e. difference between 1 days since visit vs 4 days since visit). My gut says there's a way to do this elegantly with an interaction variable but I'm not sure what that is.

  • Transforming Continuous Values into Categorical Groups (i.e. No Visit, 0-10 days, 11-20 days, etc.). Probably what I'm going with now, I'd like the ability of demarcating granular propensities with a continuous variable but this seems the most straightforward to me and I think the data would be too thin to transform days since last visit directly to categorical.

Thanks in advance for your thoughts!

EDIT 8/28: I wanted to update on the second point here what I had in mind, say I have a table of the following values. (for some reason my markdown tables aren't displaying)

Last Visit | Flag | Interaction

NULL | 0 | 0

5 | 1 | 5

NULL | 0 | 0

7 | 1 | 7

I would include the Flag and Interaction variable in my regression, with the hope that since Flag = 0 and Interaction = 0 have perfect collinearity, the values for the interaction terms coefficient estimate would be based solely on the non-null data.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

By the way, the problem you have right now is the "missing data" problem. It can be such an issue that that's one of the reasons they do redundant checkups on you for like 5 things everytime you go to the doctor regardless of whether or not you feel it's related. No one wants to deal with the NULL monster.

It can be frustrating to deal with, but there are many options available...some more complex than others....some very, very complex....tons of books written on the subject too...it's not too bad though.

First, consider what your goal is....do you really need all the people whom have never made a visit in the model? They can really cause problems (probably because they likely have other data that is NULL as well...just a guess) if they arn't relevant to the information you're really seeking and especially if they have multiple NULL values...then I'd actually just delete them and perform a complex different regression/analysis with them. I'm going to assume you feel that they are important though as otherwise you would not have asked the question here including them.

As you have already recognized by asking the question...but I'm going to reiterate:

Null values or the lack of data should never be interpreted as a 0...it's a literal lack of information...it's a blank...if you were to measure something and it was "0" then that's data....that's information. If you weren't able to measure anything, there's no data or information.

1st) I have never heard of setting their values to +/- infinity. Maybe it's a technique I'm not aware of, but seems like it mess up alot of stuff.

2nd) I would DEFINITELY have a binary variable that indicates whether or not they have ever even visited. This is super important. The regression algorithm will likely assign some sort of weight via a coeffient * (1) or (0) to balance out the people who never visited to the ones who did. In fact, you may find that it is one of your most significant variables.

Although you could separate everything into buckets like you mention (it'd be easier seemingly)...but this would probably confuse the regressional alogrithm with the bucket containing the never-shows. You're also unnecessarily eliminating information by doing summarizing unique values into buckets. You're just throwing it all away...but anyways, here's what I'd do pending a few things:

You feel you have enough other data, say, in other columns, about these no shows which you'll be using in your regression. To be honest though, if they are no shows, then you likely have many more columns with the no shows's corresponding Null value for the matching...Null cells...if you do, then at this point this is over my head and I would just move them into a separate regression you study.

Null and measurement recorded are as if they are from different worlds. I would even study the Nulls on their own...if it mattered...depends on the context. Although NULL means a lack of data for a particular parameter, it may (likely) also be related to the value of many other values you have for them...

They are special cases.

Anyways, here's the start of the fancier and likely much better approach. Depending upon the statistics software you're using, it may already have a function where it does something like this...or does it more robustly...or uses a different method.

Regressional Imputation

By doing this, you're just allowing your regressional algorithm to perform better. You're not adding any value or additional information - which is something to keep in mind. You do incur a penalty though in terms of your margin of error...because you're not using real data...it's ...simulated...

So, you set the column where the cases exist with NULL as the explanatory value to your Y. You then run a regression with all the other cases and get a formula which you could then use to predict what their NULL value "would have been" if "it was there."

This only allows your regressional algorithm to function correctly and accurately though. If you do this, you definitely afterwards then need that additional column of a binary variable representing if they are a no show or not when you perform the "final regression."

For a fancier man... Multiple Regressional Imputation

You can also get fancier by 1st) predicting the NULL values by setting them to your Y and doing as explained above...and then you have a complete data set.

You take that complete data set and run a bunch of tests over it....such as a correspondance analysis and then other types that check how "complete" your data is...I'll link you to the site I'm referencing in a second..been a while since I did one of these.

I believe you actually make multiple data sets with your new simulated values in different ways (ie. using different variables each time you perform it yielding a different prediction value)

Then after measuring all your new data sets...you run tests which explains to you an analysis of your data...you then combine all of this data testing your none null data with the simulated data and hopefully figure out any, if which, other x variables/situations you can use to most accurately predict what the value would actually be.

This gets you closer than just a single regressional imputation as you're trying to adjust for the error penalty you're incurring by studying relationships to the NULL.

Yeah, so it can get messy.

Here is a nice summary of all the techniques for treating NULL values as well as how to recognize the TYPE of null value you have and what to do with that then...https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3668100/

Improve this answer
$\endgroup$
7
  • $\begingroup$ Taal,thanks a TON for your detailed feedback here. The nulls are a population I still have to predict for given non-null characteristics. My lingering doubt is physically when I pass the values to the model, it can't permit a NULL value to be run. If I set NULL to 0 for "days_since_last_visit" and regress on: 1) Binary Null Flag (Null=0) 2) Interaction Variable: days_since_last_visit X Binary Variable Since I'd have perfect collinearity between value 0 for days and the null flag, would the coefficient generated on the interaction term ignore the resulting 0 value? (i.e. 0 * 0) $\endgroup$
    – Ben Saunders
    Commented Aug 25, 2017 at 22:58
  • 1
    $\begingroup$ @BenSaunders No, because NULL does not equal 0, unless you have VERY GOOD reason for assigning a 0 to it. I don't know the context of your data, but...I can safely assume that "days since last visit" is a very important variable for you to include since you're asking about it...so you want to get it right. The regression would then weigh everything else in a skewed fashion when generating it's result if you did this. You're breaking the natural nature of the data you have by artificially tampering with it if you just set NULL to 0. That's like blasphemy! $\endgroup$
    – Taal
    Commented Aug 26, 2017 at 1:06
  • 1
    $\begingroup$ @BenSaunders I describe above how to impute data in for the NULL values and I would use that or one of the other methods mentioned from the link at the end I provided. Imagine someone who has never been to your store (I do not know the actual context). Imagine someone who has been to your store. These people are completely different from your perspective since wherever they are visiting is important to you. Therefore, they should be treated as such. Imagine they are aliens. $\endgroup$
    – Taal
    Commented Aug 26, 2017 at 1:07
  • $\begingroup$ I get what you're saying here, and given that this population is likely significantly different I don't want to go the route of imputing data on the nulls. I understand that Null != 0, but my thinking was that if I regressed only the flag and the interaction variable, the value I set Null to would ultimately be meaningless since the interaction term would converge to a single value. I've edited my original question to more specifically describe my proposal, thanks! $\endgroup$
    – Ben Saunders
    Commented Aug 28, 2017 at 21:17
  • 1
    $\begingroup$ That does work - it's essentially @probabiltyislogic's answer. If the linear predictor is $\beta_0 + \beta_1 w + \beta_2 w x$, where $x$ is days since last visit & $w$ is your flag; then when $w=0$ you're fitting only an intercept $\beta_0$ & when $w=1$ you're fitting a new intercept & slope $(\beta_0 + \beta_1) + \beta_2 x$, & it doesn't matter what you set $x$ to when $w=0$. It's conceivable that the no. of days that would have elapsed since someone's last visit if they had visited, even though they never did, is predictive of the response, & that imputing it will improve your model. ... $\endgroup$
    – Scortchi ♦
    Commented Sep 3, 2017 at 12:06
2
$\begingroup$

This sounds like you have a predictor which is only relevant for part of the population. My advice is to (conceptually) split your population into two groups. Then write down equations for what your linear predictor should look like for each group. Then consider if these can merged back together into one model, or if you need separate models.

Using the example given, the groups are NULL vs not. Now supposing this is the only variable (for simplicity) with null values, and you have other variable, so your predictor looks like $\eta_i^{NULL} =x_i^T\beta $ or $\eta_i^{not NULL} =x_i^T\beta + z_i\alpha $ where we have $z_i$ is the days since last visit (or transformation of this), $x_i $ is the other predictors in the model. Now if I set the value of $z_i=0$ in this model - both groups have the same linear predictor. This might be good or bad, it depends on the model (though probably bad in your case). What you may also want is for the parameters to vary, so we have $\beta_{NULL} $ or possibly a subset different so you have $\eta_i^{NULL} =x_{1,i}^T\beta_{1,NULL}+x_{2,i}^T\beta_2 $ or $\eta_i^{not NULL} =x_{1,i}^T\beta_{1,not NULL}+x_{2,i}^T\beta_2 + z_i\alpha $

What this ultimately boils down to is having an indicator variable for NULL being included in your model, and deciding which other variables it should and shouldn't interact with. Once you have this, setting the NULL values to zero is appropriate. (FYI: in the above model, there is an interaction term between the null dummy and $x_1$)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.