13
$\begingroup$

When people implement permutation tests to compare a single sample against a mean (e.g., as you might do with a permutation t-test), how is the mean handled? I have seen implementations that take a mean and a sample for a permutation test, but it is unclear what they're actually doing under the hood. Is there even a meaningful way to do a permutation test (e.g., t-test) for one sample versus an assumed mean? Or, alternatively, are they just defaulting to a non-permutation test under the hood? (e.g., despite calling a permutation function or setting a permutation test flag, defaulting to a standard t-test or similar function)

In a standard two-sample permutation test, one would have two groups and randomize the assignment of labels. However, how is this handled when one "group" is an assumed mean? Obviously, an assumed mean has no sample size in and of itself. So then, what is the typical way of working the mean into a permutation format? Is the "mean" sample assumed to be a single point? A sample of equal size to the sample group? An infinitely-sized sample?

Given that an assumed mean is, well, assumed- I'd say it technically has either infinite support or whatever support you want to assume for it. However, neither of these are very useful for an actual calculation. An equal-sized sample with values all equal to the mean seems to be what is done sometimes with some tests (e.g., you just fill in the other half of the pairs with the assumed location). This makes a bit of sense, as it's the equal-length sample you'd see if your assumed mean was correct with no variance.

So my question is this: In practice, do people actually emulate permutation-test style label randomization when the second set is a mean (or similar abstract assumed value)? If so, how do people handle label randomization when they do this?

$\endgroup$
4
  • 7
    $\begingroup$ A permutation test of a specific hypothesized mean is no different from subtracting that hypothesized mean from the data and testing against a mean of zero. A paired test is discussed here; it makes the assumption that under the null the pairs have the same distribution, which implies the differences on which the subsequent one-sample test is based is assumed to be symmetric. On that basis, the signs are randomly flipped on each difference ...(ctd) $\endgroup$
    – Glen_b
    Jul 28, 2013 at 17:00
  • 1
    $\begingroup$ (ctd)... (which for a paired test is equivalent to flipping the groups labels). Well that's for a randomization test - for a full permutation test you'd do all $2^n$ possible combinations of sign-flips. If you can't assume symmetry, it's a little hard to see what you would permute - but you should still be able to conduct a bootstrap test. $\endgroup$
    – Glen_b
    Jul 28, 2013 at 17:02
  • $\begingroup$ That makes sense. But I am thinking a bit from the computational implementations that people do. If you can transform it into a sign test, do people actually bother calculating the permutations? For any sequence of length N, the full set of permutations of sign flips would be the same, no? So I would think that under the hood, people might just funnel it into a binomial test rather than manually generating the permutations that make a binomial distr. I'm mainly wondering if/when there benefits to relabeling & permuting versus using a standard test in the single-sample vs mean case. $\endgroup$
    – Namey
    Jul 28, 2013 at 19:05
  • $\begingroup$ I wasn't suggesting converting it to a sign test at all. Under the scheme I was disucssing the signs are permuted but the absolute values of the original data are retained; the $k^\text{th}$ permuted $x_i$ is $s_i^{[k]} |x_i|$ where $s$ is either $+1$ or $-1$. That is, if $x_{10}$ was 11.43 in a one sample test against a mean of zero, permuted $x_{10}$'s would all either be -11.43 or +11.43. If you ranked the absolute data first, you'd actually end up with a Wilcoxon signed rank test, so it's like the unranked (original-data) version of that. $\endgroup$
    – Glen_b
    Jul 28, 2013 at 23:28

1 Answer 1

9
$\begingroup$

Expanding Glen_b's comment into an answer

An approximate one-sample permutation test for the mean of a sample, against a null hypothesis of zero mean, is implemented by assigning random signs to the data in the sample. Non-zero null hypotheses can be tested by subtracting the desired null mean from the data.

This is easy to see in the source of the R function onetPermutation in the package DAAG. Here's an excerpt of the relevant code, with comments I've added:

function (x, nsim) {

  ## Initialize and pre-allocate

  n <- length(x)
  dbar <- mean(x)
  absx <- abs(x)  # there's actually a bug in the code; below you'll see that the function ends up re-computing abs(x) instead of using this
  z <- array(, nsim)


  ## Run the simulation    

  for (i in 1:nsim) {                             # Do nsim times:
      mn <- sample(c(-1, 1), n, replace = TRUE)   #  1. take n random draws from {-1, 1}, where n is the length of the data to be tested
      xbardash <- mean(mn * abs(x))               #  2. assign the signs to the data and put them in a temporary variable
      z[i] <- xbardash                            #  3. save the new data in an array
  }


  ## Return the p value
  # p = the fraction of fake data that is:
  #      larger than |sample mean of x|, or
  #    smaller than -|sample mean of x|

  (sum(z >= abs(dbar)) + sum(z <= -abs(dbar)))/nsim
}
$\endgroup$
2
  • 1
    $\begingroup$ What is the function of the abs in the code above? Won't the flipping of the labels be equally as random without abs? $\endgroup$ May 18, 2020 at 11:56
  • $\begingroup$ @MatthewBrett in this case I think the idea is to precisely control which data points lie on which side of 0. My intuition is that if you randomly flip the signs but the signs aren't already evenly distributed, you might not get as good of a result. $\endgroup$ Sep 2, 2022 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.