Under the Potential Outcome framework for causal analysis. Let $Y_i(W_i)$ be the potential outcome of subject $i$ if the treatment he received is $W_i\in\{0,1\}$. In reality, we only observe at most one potential outcome of each subject, i.e., if you observe $Y_{i}(1)$, then you cannot observe $Y_i(0)$.
The Fisher Sharp Null Hypothesis is the following:
$H_0: Y_i(0)=Y_i(1),~\forall~i\in\{1,2,...,N\}$.
This means: under $H_0$, the treatment has no effect for ALL subjects. Then the observed outcome in this set of subjects is merely the result of the randomization assignment procedure $\mathbf{W}$ (e.g., the randomization procedure could be each subject flip a fair coin such that head go to treatment and tail go to control group). So we can compute the EXACT distribution of any test statistic that is based on $\mathbf{W}$, then we can check what is the p-value of this observed test statistic based on its own distribution we just derived under $H_0$. And this is the permutation test.
Now here comes my question. In reality, people often use the test statistic as $T=$ average mean (or median or rank) difference between two groups. However, in my eyes, these test statistics are just some aspect of $H_0$, not $H_0$ itself. For example, under $H_0$, one can construct as many test statistic as he wants. And it could be that some of them have small p-value, and some of them have large p-value, then in this case, what should we do? Should we accept $H_0$ or reject it? For scientific research, some guys just check a lot of metrics (test statistics) when using the data, and if they just report those metrics that have small p-value, then it is not very good for the readers.
The $H_0$ is about the whole distribution of the difference between $Y(1)$ and $Y(0)$ (according to Rubin, the potential outcome, vector $Y(1)$ and $Y(0)$ are not random and only the randomization assignment procedure is the source of randomness. So here the word “distribution” means a frequency plot). But all people checked in practice is a partial property of that distribution, e.g., whether the mean or median of the 2 distribution are equal. In statistic 101, we usually facing $H_0$ itself is whether the mean of 2 groups are different. In that case, it makes sense to use difference in group mean as test statistic.
So for this Fisher Sharp $H_0$ should we always use something like $Kolmogorov-Smirnov$ test statistics?