Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
There is one variable in my data have 80% of missing data. The data is missing because of non-existence (i.e. how much bank loan the company owes). I came across an article saying that dummy variable adjustment method is the solution for this problem. Meaning that I need to transform this continuous variable to categorical?
Is this the only solution? I do not want to drop this variable as I think theoretically, it is important to my research question.
Are the data "missing" in the sense of being unknown or does it just mean there is no loan (so the loan amount is zero)? It sounds like the latter, in which case you need an additional binary dummy to indicate whether there is a loan. No transformation of the loan amount is needed (apart, perhaps, from a continuous re-expression, such as a root or started log, which might be indicated by virtue of other considerations).
This works well in a regression. A simple example is a conceptual model of the form
$$\text{dependent variable (Y) = loan amount (X) + constant.}$$
With the addition of a loan indicator ($I$), the regression model is
$$Y = \beta_I I + \beta_X X + \beta_0 + \epsilon$$
with $\epsilon$ representing random errors with zero expectations. The coefficients are interpreted as:
$\beta_0$ is the expectation of $Y$ for no-loan situations, because those are characterized by $X = 0$ and $I = 0$.
$\beta_X$ is the marginal change in $Y$ with respect to the amount of the loan ($X$).
$\beta_I + \beta_0$ is the intercept for the cases with loans.
I think you have misunderstood the suggestion of the article: mainly because the suggestion makes no sense. You would then have two problems: how to recode a variable and its values are still missing. What was probably suggested was to create a missingness indicator.
A somewhat relevant approach to handling missing data which loosely matches this description is to adjust for a missingness indicator. This is certainly a simple and easy approach, but in general it is biased. The bias can be unbounded in its badness. What this does effectively is fit two models and average their effects together: the first model is the fully conditional model, the second is a complete factor model. The fully conditional model is the complete case model in which each observation is deleted that has missing values. So it is fit on a 20% subset of the data. The second is a fit on the remaining 80% not adjusting for the missing value at all. This marginal model estimates the same effects as the full model when there is no unmeasured interaction, when the link function is collapsible, and when the data are Missing at Random (MAR). These effects are then combined by a weighted average. Even under ideal conditions, no unmeasured interactions, and missing completely at random (MCAR) data, the missing indicator approach leads to biased effects because the marginal model and conditional model estimate different effects. Even predictions are biased in this case.
A much better alternative is to just use multiple imputation. Even when the mostly-missing factor is measured at a very low prevalence, MI does a relatively good job of generating sophisticated realizations of what possible values may have been. The only necessary assumption here is MAR.
