0
$\begingroup$

I'd like to know how to impute non-normally distributed data from interval data, where the intervals differ across different individuals.

The variable I am interested in is the number of months an individual was unemployed in the three years after leaving full-time education. This variable is naturally bounded at 0-36.

I have partially observed working-histories for some individuals,though, so in some cases I can narrow this interval down. For example, I may observe an individual for 12 months and see they were unemployed in 3 of those months. Their interval can therefore be narrowed to 3-27.

These data are not normally distributed - they are heavily skewed towards zero, with a sizeable number of individuals at 36, too - due to truncation. (See picture below.)

Given this I think I should use predictive mean matching. I'd like to know if anyone knows of any method/software where the candidate pool of donors can be restricted based on the individuals' specific interval. Otherwise, I'm likely to impute values which I know to be false - i.e. outside the known bounds.

I know of programs which can use different bounds for different individuals, but these assume normally distributed data (e.g. ice package in Stata) or just truncate values afterwards if the imputed value is outside bounds (e.g. mice in R).

Does anyone have any suggestions please?

Thanks.

enter image description here

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Did you actually try PMM ? In my experience it does not impute values outside the relevant range. An alternative is hot-deck imputation. $\endgroup$ – Robert Long Apr 12 '19 at 19:25
  • $\begingroup$ @RobertLong that is essentially a plausible answer. Predictive mean matching is a cross between propensity score matching and imputation - you basically match by random draw from the k observations most similar to the one being imputed. Note that by default, k = 1 in Stata, and you can and almost certainly should change that to 5 or so. $\endgroup$ – Weiwen Ng Apr 12 '19 at 19:45
  • $\begingroup$ @RobertLong, I haven't tried it yet as need to get covariates together. Have you tried PMM in this setting where each individual has their own range? $\endgroup$ – lwri Apr 13 '19 at 7:13
  • $\begingroup$ There is a broader statistical question of whether you can even model that kind of data with a standard modeling function, let alone perform imputation on it. As you said, there exist methods for modeling when the residual are normally distributed (doesn't have to be the data that are normally distributed), such as truncreg and intreg in Stata, which are compatible with mi impute chained. Perhaps ask another question about how to model that data. Then you can attempt to customize a MI software with that model. $\endgroup$ – Noah Apr 26 '19 at 3:00
0
$\begingroup$

@RobertLong commented first, but hasn't fleshed this out into an answer. Predictive mean matching is a bit like propensity scoring. I'm simplifying the description in the link, but for each observation with missing $x$, you find the $k$ complete data observations that should be most similar to the one with missing data, then impute one of the $k$ nearest neighbors' complete data observations, then repeat the process (note, it's more complicated that that and involves an additional stochastic step I didn't describe; read the link).

You're drawing imputations from actual data, sort of like you would in bootstrapping. So, the imputed data are going to match the characteristics of the observed data.

Since you're using Stata, one issue you need to be aware of is that the default setting has $k = 1$, which Allison (the link above) argues is much too low and produces too little variability, and that you should set $k =$ 5 or 10.

In Stata, the syntax would be

mi impute pmm Status3_M = x1 x2 x3 ... , knn(5)

Or, if doing as part of a larger imputation by chained equations,

mi imputed chained (pmm, knn(5)) Status3_M (regress) ... (whatever else) ... = x1 x2 x3

Default $k$s vary from program to program, but SPSS apparently shares Stata's default. SAS and R's mice default to $k=5$.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This doesn't address OPs problem. It's true PMM bounds imputed values at the sample bounds, but OP wants to bound imputed values at specific, different values for each individual. PMM would rely on the bounds of other individuals; for example, if the k nearest neighbors had bounds of 30 but the individual in question had a bound of 20, the individual's bound would not be respected by KNN. $\endgroup$ – Noah Apr 13 '19 at 20:25
  • $\begingroup$ Thank you for the replies. What I am looking for is a PMM algorithm that adds in extra step that donor pool is restricted to observed values which are within the known interval for the missing value. By example, with k=1, let's say we have one missing observation and three observed values. Predicted values are 4, 1, 5, 6 (missing and observed, respectively). Observed values are 2, 4, 7, and known interval for missing is 1-3. First observed value with be taken (2) as only one in known interval, even though the predicted score for this observation is furthest away. $\endgroup$ – lwri Apr 15 '19 at 0:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.