Hypothesis testing for non-normal data

Question

I'm measuring the error produced by operators executing planar cuts under different guiding systems. I define a target plane and then measure the euclidean distance from uniformly sampled points in the target plane to the executed plane. Since the target plane is my coordinate system reference, I define it as the plane $xz$ in $\mathbb{R}^3$. The executed plane $E$ is defined as $ax + by + cz + d=0$ with $b \neq 0$ and, given that I'm sampling points from $xz$ and projecting them into $E$, then $b$ could never be $0$. Then $E$ could be rewriten as $y = -\frac{a}{b}x - -\frac{c}{b}z -\frac{d}{b}$ with $x \sim \mathcal{U}(x_{min}, x_{max})$ and $z \sim \mathcal{U}(z_{min}, z_{max})$. Of course, the cutting tool also produces some (maybe normal) noise that affects $y$. The random variable $y$ is not normally distributed (I've tested it and I've plotted some qq-plots of it).

I have 2 different guiding systems and I have to test if they are significantly different wrt the above described measurement method. Which method do you recommend me to use to test my hypothesis?

Thanks!

Answer 1

You suggest the possibility of a Kruskal-Wallis test, and it's quite possible that it's suitable.

Indeed, the criterion you mention in comments - "closer to zero more often" - directly suggests doing a Kruskal Wallis on the absolute deviation from zero. The two-sample version (a Wilcoxon-Mann-Whitney test) in fact has a direct interpretation in a "smaller more often" sense -- indeed in that case you can scale the U-statistic to an estimate of that probability (by dividing the U statistic by the total number of pairwise comparisons - i.e. its maximum possible value).

If your interest is instead more in some measure of location shift, then with an additional assumption (that the distributions are identical in shape, aside from possible location shift), the Kruskal-Wallis may still be a good choice. Further, in the two-sample case, there's a location-shift estimate that's readily available (one can also obtain acceptance regions - akin to confidence ellipsoids in ANOVA - for the whole set of location-shifts)

Those two possible ways to view the Wilcoxon-Mann-Whitney are discussed in more detail here.

The Kruskal-Wallis is similar, but there's not an exact correspondence because the K-W statistic doesn't see non-transitive differences but pairwise WMW comparisons can (See the discussion in item (3) in this answer); usually the second component is relatively small, but sometimes can be important.