I have run an experiment in which I've measured some metric X, and collected an associated attribute attr's value if and only if the value of X exceeds some threshold t. If a case's X value doesn't exceed t, then attr is not captured. Here's what my data look like:

ID  attr_val  no_attr  outcome
 0       NaN        1        0
 1       NaN        1        1
 2         0        0        1
 3         4        0        1
 4         2        0        1
 5         1        0        0

Where ID is a unique identifier for each case, no_attr indicates whether the attr value was captured for that particular case (i.e. whether the X value exceeded t), and the binary outcome outcome is shown.

Now, I want to predict outcome based on the attribute value if a case's X exceeded t, and I also want to measure whether a case's X not exceeding t is predictive, as well.

In order to keep IDs 0 and 1 in the model, attr_val will need to be populated with some value, not just nulls. But I don't really feel comfortable filling in 0, for example, because IDs 0 and 1 didn't have a chance to give their attr_val because their X values didn't exceed t. However, this X-exceeding-t criterion is very important to my experiment, so I can't just take the attr_vals for rows 0 and 1 anyway.

Running a logistic regression of the form outcome ~ attr_val + no_attr currently would make the design matrix singular, as just zeroes are included for no_attr if I haven't filled in any nulls in attr_val. Is the right approach here to augment no_attr by 1 so we're not multiplying by zeroes down the line? Or is there a better way to encode this problem?

  • $\begingroup$ Do you not want to use $X$ itself as part of the the predictor, since you have measured it? Or has the value of $X$ been discarded? Are the attribute values numeric or are they just arbitrary categorical labels? Does an attribute value of 4 mean that $X$ was higher than when the attribute is 1 or 2? $\endgroup$ – Gordon Smyth Dec 18 '17 at 3:29
  • $\begingroup$ The value of X has been discarded. The attribute values are continuous in [0, inf). To your last question, no; the inverse of the no_attr (has_attr?) is essentially a link for a particular case's being able to have a value for attr_val. $\endgroup$ – blacksite Dec 18 '17 at 13:35

Question of this sort are easily resolved by writing down your model.

Suppose you did use some number, say $a$, to represent the missing values of attr_val. Then your model would be in the form

$$\Pr(\text{outcome}) = h(\beta_0 + \beta_{\text{attr_val}} [\text{attr_val}] + \beta_{\text{no_attr}} [\text{no_attr}])$$

for your chosen link function $h$. Since the probabilities depend on the argument of $h$, we will focus on its values.

Consider two cases:

  1. attr_val is not missing. Then no_attr is zero and $$\Pr(\text{outcome}) = h(\beta_0 + \beta_{\text{attr_val}} [\text{attr_val}] ).$$

  2. attr_val is missing. Then attr_val is set to the value $a$, no_attr equals $1$, and the formula becomes $$\Pr(\text{outcome}) = h(\beta_0 + \beta_{\text{attr_val}} a + \beta_{\text{no_attr}} ).$$

You may interpret $(1)$ as a model for the output when attr_val is not missing and $(2)$ determines the fit when attr_val is missing. If you conveniently choose $a=0$, this simplifies further: $\beta_0 + \beta_{\text{no_attr}}$ determines the fit in such cases, allowing you to interpret $\beta_{\text{no_attr}}$ as a change in the intercept $\beta_0$ attributed to the circumstances $X\le t$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.