Appropriate statistical test for paired survey data?

Question

I'm somewhat new to statistics, so forgive me for a rather elementary question. I'm not looking for someone to do any sort of in-depth explanation, rather would just like to be pointed in the right direction. I'm changing the specifics of the survey details to simplify but still represent the general task.

I have survey data which asks each respondent a question about their behavior during one year (say, 2019) and their behavior during another year (e.g., 2020). It is a simple integer, let's say for an example that it's the number of vacations taken in each year.

My overall objective is to determine the change in the sum of vacations in this population between the two years, and try and disprove the hypothesis that the sum of vacations is the same for the two years in this population. The population of interest is small (about 5000 people), and I managed to sample approximately 250. In order to represent the whole population, I am using scaling factors to scale up the survey data (the scaling factors are specific to demographics, but for simplicity let's say it's an entirely homogeneous sample relative to the population, so I use a scaling factor of x20).

It seems I have paired data, since the same respondents are answering the question for two separate time periods. What statistical test is appropriate to determine a confidence interval around my difference of sums? In the end, the goal is to say something like: (1) there is a statistically significant (alpha = .05) difference between the sum of vacations in 2019 vs 2020, (2) we estimate this difference in the sums to be X +/- CI.

Answer 1

Perhaps the following data, simulated in R and summarized below, are somewhat similar to your data. After brief mentions of various confidence intervals, I show how to find a nonparametric bootstrap for these $n = 250$ hypothetical differences in number of vacations from one year to the next.

summary(d)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 -4.000   0.000   2.000   2.236   4.000  11.000 
 boxplot(d, col="skyblue2", pch=19, horizontal=T, notch=T)

Boxplot Notches. The 'notches' in the sides of the box in the boxplot provide a nonparametric confidence interval $(1.6, 2.4)$ about the median $2$ of the data. [This CI is mainly intended for use in comparing two boxplots: If notches don't overlap, then the samples used to make the boxplots may have significantly different medians. A little experimentation with normal samples of moderate size (a few hundred) shows that the notch-intervals correspond roughly with 95% t intervals. For further details, please see @Glen_b's answer here.]

boxplot.stats(d)$conf
[1] 1.600288 2.399712

Wilcoxon 95% nonparametric CI. In R, the procedure wilcox.test does a one-sample Wilcoxon signed-rank test and shows a confidence interval upon request (parameter conf.int=T.) The resulting 95% nonparametric CI is $(2.5, 3.0).$ [For more about this confidence interval (related to the "pseudo-median") see @Glen_b's answer here.]

wilcox.test(d, conf.int=T)

        Wilcoxon signed rank test with continuity correction

data:  d
V = 20844, p-value < 2.2e-16
alternative hypothesis: true location is not equal to 0
95 percent confidence interval:
 2.499967 2.999988
sample estimates:
(pseudo)median 
      2.500049 

Nonparametric 95% quantile bootstrap CI. A simple 95% quantile nonparametric bootstrap CI for the population mean, is found by repeatedly re-sampling samples of size $n = 250$ with replacement, finding the mean of each re-sample, and then finding quantiles $.025$ and $.974$ of the resulting bootstrap distribution of sample means. The resulting CI for my data above is $(1.91, 2.56).$ [Of the three types of CIs discussed on this page, this may be the easiest to interpret: it is clear what is estimated (population mean difference) and at what level of confidence (95%). This is one of many possible styles of nonparametric confidence interval.]

set.seed(2021)
a.re = replicate(5000, mean(sample(d, 250, rep=T)))
q = quantile(a.re, c(.025, .975));  q
  2.5% 97.5% 
 1.908 2.560 

hdr = "Nonparametric Bootstrap Distribution of Means"
hist(a.re, prob=T, col="skyblue2", main=hdr)
 abline(v = q, col="red", lwd=2, lty="dotted")

---

Notes:

(1) For "completeness" I mention the 95% t CI for normal data. Our differences d are far from normal, according to a Shapiro-Wilk test:

shapiro.test(d)$p.val
[1] 0.0001377481

Some might argue that sample size $n = 250$ is large enough to trust in the legendary robustness of t methods. I would not use it. For the record, the resulting 95% CI is given below:

t.test(d)$conf.int
[1] 1.906048 2.565952
attr(,"conf.level")
[1] 0.95

(2) Fake data for the illustrations on this page were simulated by the R program below:

set.seed(319)
x.19 = rpois(250, 4);  x.20 = rpois(250, 2)
d = x.19 - x.20