1
$\begingroup$

I'm working with dataframes (one for each of 185 locations) that shows sums of occurrences for each calendar date. There are no 0 values for occurrences in the entire dataset. There are several missing dates.

Below is a dataframe I created which shows the mean daily occurrences for some example locations. CI = 95% confidence interval of expected occurrences of 0, N/A = number of missing entries, ENTRIES = number of actual entries + number of missing. My null hypothesis is that all the missing dates are actually 0 counts.

    MEAN VALUE  CI      N/A   ENTRIES   P-VALUE
0   10.306675   0-1     1       1589    1.000000
1   5.416826    1-14    20      1589    0.141214
2   8.081301    0-0     2       248     0.088841
3   9.044795    0-1     4       1589    0.077208
4   7.531390    0-0     3       449     0.062347
5   7.720607    0-3     7       1589    0.043126
6   10.056911   0-0     2       248     0.026038
7   8.661017    0-1     4       1066    0.023237
8   6.040712    0-9     17      1589    0.021933
9   6.546326    0-6     14      1579    0.007288
10  9.160428    0-0     3       377     0.006710
11  8.538219    0-1     6       1589    0.005819
12  4.893248    4-20    34      1589    0.005412
13  3.400536    34-71   96      1589    0.004372
14  29.995498   0-0     34      1589    0.000000

The issue I'm having is how to impute data for the missing dates. For very low p-value, I believe imputing an average or forward fill for these locations' data would be the correct choice. Locations with a high p-value, such as the first location, I would believe imputing the missing data as a 0 would be appropriate.

However, when it comes to locations as such 13, I'm stuck. Some 0 counts should be expected, as reported by the confidence interval, but there are several more actual N/A then should be expected, which makes me think imputing all missing dates as either an average, or all as 0 would be incorrect.

Would it make sense for this location, to impute some of the missing dates as 0, and the rest as an average? Sorry if this has been asked. I'm a novice and not sure on my wording, and choices. Thanks for any help!


Edit: The p-value is from a binomial test. I counted N/A as the number of 'successes', ENTRIES as number of trials, and p= success rate from 97.5th percentile of CI.

I got my confidence interval from 10,000 Bernoulli trials of drawing 1,000 samples from a Poisson distribution (using MEAN VALUE as the event rate,lambda). Would count each 0 as success. This confidence interval (expressed as success rates) would then be multiplied by ENTRIES for each location to give the range of expected 0 count days.

Unsure if I used a suitable model, still learning.

$\endgroup$
  • 2
    $\begingroup$ Could you please tell us how the values in the "P-VALUE" column are determined? What is your model and what test are you applying? $\endgroup$ – whuber Nov 25 '18 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.