1
$\begingroup$

In my final data set I have a high proportion of missing data (1/4 are NAs, 35/52 variables have NAs), because of my codebook design: I only coded features (how things were discussed) that occurred in the text and I had a few conditional questions (e.g. Did they perceive a threat? What was the nature of the threat?). To analyze the data, I want to use classification trees (CART). However, the high proportion of missing values make excluding observations with NAs or imputation impossible.

I believe I should probably deal with that problem on a conceptual level and came up with two tactics:

  1. Do not take the values coded -77 other, -88 uncertain, or -99 not applicable as missing values (like usual) but as their own values (also because they are different NAs). Thus, they would become part of the model. Edit: And if I include them, how would that work when the variable is best conceived as ordinal?

  2. Have default values instead of coding something as -99 not applicable. For conditional question, for instance, "What was the nature of the threat?", a questioned conditioned on a "yes" in "Did they perceive a threat?", would get a category of "no threat". Some normal questions could also receive a default value based on the assumption that another value would have been mentioned in the text because it would affect the outcome (a decision by the body).

Are any of those two tactics an adequate way of dealing with the problem? Are there better ways?

Improve this question
$\endgroup$
  • $\begingroup$ hello - I notice you haven’t referred to any of the sources that exist on strategies for addressing missing data. Any reason? $\endgroup$ – rolando2 Jun 27 '20 at 15:22
  • $\begingroup$ @rolando2 One potential source is here: stat.columbia.edu/~gelman/arm/missing.pdf (talks about imputation, excluding these observations, and - in 25.3 - about "indicator variables") $\endgroup$ – Quanttek Jun 28 '20 at 12:06
  • $\begingroup$ If you have read multiple such sources and have run into a specific stumbling block, that would be a good starting point for a question posted here. Your current post is quite general and because of that I'm afraid it will be voted to be closed. $\endgroup$ – rolando2 Jun 28 '20 at 18:26
  • $\begingroup$ You have a lot of missing data. How is the missing data intentional? In my experience, missing data is due to 1. measurement failure where a system failure occurred (e.g. patient temperature not taken, software failed) or 2. in surveys typically, people refuse to answer a question (income!). The only situation I can think of with intentionally missing data is in choice analysis - which has specific methods for dealing with the (intentionally) missing values. You need to add more information to your question, as it is currently missing important information. $\endgroup$ – Michelle Jul 3 '20 at 23:40
  • $\begingroup$ "Are there better ways?" - Modern forms of imputation are better ways. E.g., Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annu. Rev. Psychol. 60:549-576. personal.psu.edu/jxb14/M554/articles/Graham2009.pdf $\endgroup$ – rolando2 Jul 4 '20 at 1:34