I've a set of measurements arranged in a matrix $A \in R^{n \times m}$, where $A_{i,j}$ is the value of a function for sample $i$ under transformation $t$, that is $A_{i,j} = f( T_j (x_i))$.

I wish to measure if $f$ is invariant to $T_1, \dots T_m$, given $n$ samples $x_i$.

One way to indirectly test this is to perform 1-way ANOVA. If the null is rejected, we can interpret that result as $f$ not being invariant.

However, given the nature of $f$, the most natural prior would be to assume $f$ is not invariant (different means), and test for invariance (equivalence of means).

@whuber in the comments mentionts TOST as a way to perform equivalence testing, and @Alexis points out several articles that generalize TOST for multiple comparisons, similar to ANOVA. I think this could work

In both cases, these approaches disregard the relationship between values of the same row of $A$. Another ad-hoc measure I've considered is just the mean of the variance of the rows, ie $V_m = Mean( Var(A[1,:]), \dots, Var(A[n,:]) )$. If this mean is 0, then $f$ is clearly invariant but I'm not sure how to perform statistical test on $V_m$.

  • 2
    $\begingroup$ What is the point of using "all means are different" as null hypothesis? This is a very vague null hypothesis. It is much more common to have 'the absence of some effect' to be considered as the 'null hypothesis'. This is most often a point and not a whole range. What underlying theory do you have that is the subject of this test? $\endgroup$ Commented Oct 8, 2019 at 20:01
  • 5
    $\begingroup$ This is a useless and ineffective null: useless because it sets no bounds on how much the means could differ (would differences of one part per trillion count as real differences, for instance?) and ineffective because it does not determine a distribution of any statistic for use in testing. For related questions--which might help you reformulate this one--search our site for TOST. $\endgroup$
    – whuber
    Commented Oct 8, 2019 at 20:42
  • 3
    $\begingroup$ You may wish to read Wellek, S. (2010). Chapter 7: Multisample tests for equivalence. In Testing Statistical Hypotheses of Equivalence and Noninferiority (Second). Chapman and Hall/CRC Press. It's more complex than TOST, but gives a manner of forming the appropriate omnibus null hypothesis for either TOST or UMP. Some more references of potential relevance follow: $\endgroup$
    – Alexis
    Commented Oct 8, 2019 at 20:55
  • 2
    $\begingroup$ Mara, C. A. (2013). Testing for Equivalence of Group Variances (Ph.D. Thesis). Graduate Program in Psychology, York University, Toronto, ON. $\endgroup$
    – Alexis
    Commented Oct 8, 2019 at 20:56
  • 1
    $\begingroup$ Mara, C., & Cribbie, R. A. (2017). Equivalence of Population Variances: Synchronizing the Objective and Analysis. The Journal of Experimental Education, 86(3), 442–457. $\endgroup$
    – Alexis
    Commented Oct 8, 2019 at 20:57


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.