I have 2 columns of ~18000 observations. First column contains a value that shows the delay of a filed document (in days) of year 2017. For example: if the document had to be filed on May 5th 2017, but was filed on May 7th 2017, the value for that observation would be "2". The second column contains the same observation for the year 2018. The document needs to be filed annually.

Quirks of the data:

  1. If the document was filed in a timely manner, the observation is "0";
  2. If the document had to be filed, but wasn't at all, the observation is "Q";
  3. If the document didn't have to be filed and wasn't filed, the observation is empty;
  4. The rest of the observations are natural numbers.

Question: Assume an action was taken to increase compliance of filing this document. Is it possible to measure the effect of this action?

So far I've divided the 2017 column into 4 subsets, filtered by the observation type (0, Q, Empty, Natural) and made a matrix, showing the distribution of the 2018 column in each of these subsets. Is there a better test I could do for this problem?



Provided an intervention occurred on the cusp of 2017 and 2018 (you could probably live if it was within ~1 month of the New Year's date), you could:

  1. Explore the distributions and compare the means using an independent t-test (if the distributions are roughly normal), or the medians using a Wilcoxon-Mann-Whitney U-test (if the distributions are non-normal). This will treat each observation in each year independently, and will use all available data even if either 2017 or 2018 value is missing from one of the rows.

  2. If the data in the two columns are paired, i.e. if the 2017 and 2018 observations in each row belong to the same entity (it is not clear from the question if this is the case), you could conduct a paired t-test, which is a much more powerful version, but only on the condition that the distribution of the differences is roughly normal. You could check this by subtracting 2017 values from 2018 values (or vice versa, as it doesn't matter), and plotting a histogram. As you can probably see, the paired t-test will omit all rows where either a 2017 or a 2018 value is missing, legitimately (EMPTY) or illegitimately (Q), as the difference cannot be calculated in these cases.

  3. It sounds like the illegitimately missing values (Qs) are of an interest of themselves, as their rate of occurrence might indicate the level of compliance. You could supplement 1/2 (above) with the comparison of the proportion of Qs in 2017 vs 2018. It would probably make sense to look at the proportion of Qs among active entities only, i.e. Qs/(Qs+Ns+0s), excluding empty values. R's prop.test() is a common way of doing this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.