4
$\begingroup$

I have a regression problem where the independent variables are all factors (categorical). I've been looking at the literature on missing data, and so far it all seems concerned with missing training data. I was wondering if there is a standard way of dealing with missing data in the prediction set. That is, you have all the data you need to train, but then you need to be able to make a prediction with only partial data. This must have been a studied problem.

My initial thought is to use an average of the dummy encoded variables, according to how common they are. As a quick example, say we have a three level factor dummy encoded as

level 1: [1 0]
level 2: [0 1]
level 3: [0 0]

Say level $i$ occurs fraction $f_i$ of the time in the training data (so $\sum_i{f_i}=1$), and the regression has the two coefficients $\beta_1$ and $\beta_2$.

Then a missing value in this factor might be estimated as: $$ \beta_1*f_1 + \beta_2*f_2 + 0*f_3 $$ But given that the "default" level encoding are shared across factors, I'm not sure I'm handling level 3 correctly in this case.

$\endgroup$
4
$\begingroup$

(I'll let someone else address the estimation of the missing data. You may want to directly model the probability that the observation is each level of the unknown factor using knowledge of other covariate values, and possibly outside information, e.g., priors etc. There are strategies such as propensity scores that you might be able to use for this type of thing. However, at first glance your approach looks reasonable to me.)

One note is that I can't tell from your description if you are weighting by raw frequencies. If so, you want to divide these by $N$ to get the marginal probabilities instead.

You are right that you are not handling level 3 correctly. The coding scheme that you use in your question set up is known as reference level coding. To use this approach correctly, you need to have an intercept (i.e., $\beta_0$), which estimates the mean of level 3. I suspect you do have such, even though you didn't list it. In this case, you would just add the intercept to your final equation. That is: $$ \beta_0\!*\!f_3 + \beta_1\!*\!f_1 + \beta_2\!*\!f_2 $$ Note that you are multiplying the intercept (which encodes the reference level) by the marginal probability that the observation is actually the reference level.

$\endgroup$
  • $\begingroup$ I do have a common intercept $\beta_0$, but I have a number of factors in the model. I was just showing one factor in the example. Is it possible to suppress the intercept with more than one factor? I thought that you could do it for one, but then it broke down with multiple. That would fix my problem, as it would allocate the overall intercept to each factor. $\endgroup$ – cschmidt Oct 10 '13 at 16:02
  • $\begingroup$ You can use your entire regression equation, ie w/ the intercept & the other factors & continuous covariates, etc. The only difference is that you are multiplying the parameters related to the unknown factor by the marginal probabilities that the observation is each possible level of the unknown factor. $\endgroup$ – gung - Reinstate Monica Oct 10 '13 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.