4
$\begingroup$

After some detective work in my data sources, I realized the reasons for my previously reported 98% of missing data ratio. After implementing some data collection code fixes, the current missing data ratio is around 35%. A multiple imputation (MI) procedure, implemented by using Amelia R package, has decreased the missing data ratio to around 30%. However, the MI procedure introduced negative values into the imputed data set, as shown in the output below. My question is twofold:

A) Is it normal for MI to introduce negative values and is it worth doing in my case, considering not a very significant decrease in missing data ratio?

B) Is this imputed data set still a valid representation of the original one and, thus, a population, so that it can be used in further analysis, in particular CFA and SEM?

The output of the MI procedure (with my code output - summary() and describe()):

Performing Multiple Imputation (MI)...Completed.

MI Results:
===========

Amelia output with 5 imputed datasets.
Return code:  1
Message:  Normal EM convergence.

Chain Lengths:
--------------
Imputation 1:  5
Imputation 2:  5
Imputation 3:  5
Imputation 4:  5
Imputation 5:  5

Rows after Listwise Deletion:  58355
Rows after Imputation:  69275
Patterns of missingness in the data:  8

Fraction Missing for original variables:
-----------------------------------------

                        Fraction Missing
Project License                  0.35667
License Restrictiveness          0.35667
Development Stage                0.32262
Project Maturity                 0.32262
User Community Size              0.36935

Data before MI:
===============

 Project License License Restrictiveness Development Stage Project Maturity
 Min.   : 1.00   Min.   :1.0             Min.   :1.00      Min.   :1.0
 1st Qu.: 3.00   1st Qu.:1.0             1st Qu.:3.00      1st Qu.:1.0
 Median : 3.00   Median :2.0             Median :5.00      Median :3.0
 Mean   : 8.53   Mean   :1.9             Mean   :4.16      Mean   :2.5
 3rd Qu.:10.00   3rd Qu.:3.0             3rd Qu.:5.00      3rd Qu.:4.0
 Max.   :82.00   Max.   :4.0             Max.   :7.00      Max.   :4.0
 NA's   :35667   NA's   :35667           NA's   :32262     NA's   :32262
 User Community Size
 Min.   :       1
 1st Qu.:     629
 Median :    5523
 Mean   :  157872
 3rd Qu.:   45146
 Max.   :21920927
 NA's   :36935

More detailed summary statistics:
=================================

                        vars     n      mean        sd median  trimmed     mad
Project License            1 64333      8.53     14.42      3     5.06    1.48
License Restrictiveness    2 64333      1.90      1.01      2     1.79    1.48
Development Stage          3 67738      4.16      1.40      5     4.23    1.48
Project Maturity           4 67738      2.50      1.44      3     2.50    1.48
User Community Size        5 63065 157872.48 665188.73   5523 18462.72 8000.11
                        min      max    range  skew kurtosis      se
Project License           1       82       81  3.57    13.10    0.06
License Restrictiveness   1        4        3  0.62    -0.97    0.00
Development Stage         1        7        6 -0.57    -0.31    0.01
Project Maturity          1        4        3 -0.02    -1.92    0.01
User Community Size       1 21920927 21920926 10.38   225.89 2648.81

Data after MI:
==============

 Project License   License Restrictiveness Development Stage Project Maturity
 Min.   :-51.316   Min.   :-2.669          Min.   :-0.294    Min.   :-3.091
 1st Qu.:  3.000   1st Qu.: 1.000          1st Qu.: 3.000    1st Qu.: 1.000
 Median :  3.000   Median : 2.000          Median : 5.000    Median : 3.000
 Mean   :  8.589   Mean   : 1.911          Mean   : 4.154    Mean   : 2.501
 3rd Qu.: 10.000   3rd Qu.: 3.000          3rd Qu.: 5.000    3rd Qu.: 4.000
 Max.   : 82.000   Max.   : 5.688          Max.   : 8.094    Max.   : 6.966
 NA's   :30725     NA's   :30725           NA's   :30725     NA's   :30725
 User Community Size
 Min.   :-2523277
 1st Qu.:     447
 Median :    5510
 Mean   :  138006
 3rd Qu.:   60657
 Max.   :21920927
 NA's   :30725

More detailed summary statistics:
=================================

                        vars     n      mean        sd median  trimmed     mad
Project License            1 69275      8.59     14.42      3     5.41    1.48
License Restrictiveness    2 69275      1.91      1.01      2     1.80    1.48
Development Stage          3 69275      4.15      1.40      5     4.23    1.48
Project Maturity           4 69275      2.50      1.44      3     2.50    1.48
User Community Size        5 69275 138005.57 669718.66   5510 23136.72 8108.34
                                min         max       range  skew kurtosis
Project License              -51.32       82.00      133.32  3.31    12.09
License Restrictiveness       -2.67        5.69        8.36  0.58    -0.91
Development Stage             -0.29        8.09        8.39 -0.56    -0.31
Project Maturity              -3.09        6.97       10.06 -0.02    -1.88
User Community Size     -2523276.98 21920927.00 24444203.98  9.26   201.22
                             se
Project License            0.05
License Restrictiveness    0.00
Development Stage          0.01
Project Maturity           0.01
User Community Size     2544.51

Saving imputed data... Done.
$\endgroup$
  • 2
    $\begingroup$ Presumably negative values don't make sense for these variables - otherwise there's no problem at all. $\endgroup$ – Scortchi - Reinstate Monica Sep 24 '14 at 12:23
  • 4
    $\begingroup$ See page 27 for a discussing of logical bounds: cran.r-project.org/web/packages/Amelia/vignettes/amelia.pdf $\endgroup$ – zkurtz Sep 24 '14 at 12:25
  • 1
    $\begingroup$ @zkurtz: I can't make any sense of the last part of the quoted sentence. $\endgroup$ – Scortchi - Reinstate Monica Sep 24 '14 at 12:36
  • 1
    $\begingroup$ @Scortchi It certainly seems to fly in the face of the otherwise Bayesian flavor of Amelia's procedures: if we know a value can't be negative, there is zero probability below zero! $\endgroup$ – Sycorax says Reinstate Monica Sep 24 '14 at 12:41
  • 2
    $\begingroup$ @user777: There's an argument here that it's only terribly important that the mean across imputations falls within the logical bounds. $\endgroup$ – Scortchi - Reinstate Monica Sep 24 '14 at 12:57
8
$\begingroup$

If negative values are impossible for these features, you should impose a constraint/prior on the imputation procedure. The Amelia II documentation provides extensive explanation of how to go about this from many different angles. You can impose a boundary, or a prior with most of is mass far from the boundary, or transform the data, or...

To answer your questions: Yes, it's normal for MI to introduce negative values in some circumstances, but if missing values are impossible, you haven't actually improved anything about your data. You'll have to correct the imputation procedure itself.

I can't speak for CFA or SEM since I'm unfamiliar with those methods. But if you believe that the imputed data contain impossible values, I think that any analysis will be tainted by that. Garbage in, garbage out.

I believe that you still have missing values after imputation because there are no non-missing values for those observations. That is, the entire row is NA. So Amelia has no basis for making a guess about what values might be plausible. One clue is that the same number of observations are missing in each column after imputation. Basically, you can't get anything for free: you'll have to collect this data in order to study it.

I'm not sure what you mean by MI being justified. What do you have to justify? On what grounds? To whom? On the one hand, you now have a slightly larger sample size, and provided that the assumptions of imputation are satisfied, reason to believe that the imputed data will mitigate the problems of bias that listwise deletion can introduce. On the other hand, when 30% of your data is just rows of NA, I'd be concerned that your data collection method leaves much to be desired, and that a sophisticated procedure won't salvage it. Perhaps these rows are missing completely at random, you can delete them and move on with your life. But if their missingness is not independent of what you intend to study, you're in a bad spot.

$\endgroup$
  • $\begingroup$ I greatly appreciate your answer as well as previous comments. Upvoting and accepting it. In regard to missing values in my study, I believe that they are possible, in a sense that CFA and SEM can handle them. To be more sure, I will investigate this issue further. But, nobody directly answered my question about whether a 5% reduction in missing values ratio is worth the trouble. $\endgroup$ – Aleksandr Blekh Sep 24 '14 at 13:09
  • $\begingroup$ I'm confused by your comment. I've never said anything about 0 missing values. That part of my question was regarding a feasibility to perform MI with its potential disadvantages (like negative values, etc.) in order to gain just 5% (35% - 30%) reduction in missing values ratio. $\endgroup$ – Aleksandr Blekh Sep 24 '14 at 13:21
  • $\begingroup$ I'm afraid I don't know what that means. A Google search for the phrase "missing data ratio" only brings up this question. Could you clarify your meaning? I interpreted it to mean the percent of missing observations, but you've indicated that is not your meaning. $\endgroup$ – Sycorax says Reinstate Monica Sep 24 '14 at 13:22
  • $\begingroup$ You don't know what? Are you referring to what I call "missing values ratio"? $\endgroup$ – Aleksandr Blekh Sep 24 '14 at 13:26
  • 1
    $\begingroup$ Just discovered your update - sometimes SE notifications are a mystery... Let me try to clarify what I mean. I understand that, if the data is not MCAR, things are not good. However, I have reasons to believe that this is a case of MCAR. In addition to my hypothesis (based on domain knowledge), I want to test data on being MCAR, using two tests (as described in the end of my question here: stats.stackexchange.com/q/116025/31372). $\endgroup$ – Aleksandr Blekh Sep 24 '14 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.