# Contradictory p-value and confidence interval in permutation test

I am using the MICOM procedure to test for measurement invariance among groups in PLS SEM modelling [1]. This procedure consists of 3 steps, where in step 3 one tests for equality of composite means values using a permutation test: "we apply PLS to obtain construct scores, using the pooled data. We then examine whether the mean values (and variances) between the construct scores of the observations of the first group and the construct scores of the observations of the second group differ from each other. Full measurement invariance would imply that (both) differences equal zero (or are at least non-significant)." [1]

Using SmartPLS, the results for one of my latent variables in step 3 of the MICOM procedure are as follows:

EDIT: Changed CI 5% to CI 95% (it was a typo). See first comment and response

Orig. mean diff.: -0.264; Mean - Permutation Mean Diff.: 0.001; CI 95%: [-0.056, 0.056]; p-value: 0.000

(OBS! Orig. mean diff. is calculated for the original groups, using the observed data. More specifically, for a given LV, its composite scores are calculated for all observations (i.e. using all the data, from both groups) using PLS-SEM. Then, for each group, the mean of the composite scores corresponding to observations within that group is calculated. Orig. mean diff. is the difference of the calculated means.)

So, according to the p-value, the difference of means is statistically significant, so there is no full measurement invariance. However, in page 416 of the MICOM paper [1], the authors say: "If the confidence intervals of differences in mean values (and logarithms of variances) between the construct scores of the first and second group include zero, the researcher can assume that the composite mean values (and variances) are equal. In this case, full measurement invariance has been established"

In my case, the CI includes zero, so according to the paper we could claim that the composite mean values are equal. This is in contradiction to what the p-value says. So how should I interpret this result?

On a side note, how is the CI calculated here? When reading literature about permutation tests, I always read about calculating p-values, but no CIs.

[1] Henseler, J., Ringle, C.M. and Sarstedt, M. (forthcoming), “Testing measurement invariance of composites using partial least squares”, International Marketing Review.

• Can you post a reproducible example ? One thing that seems strange is that your quoted output says CI 5% – Robert Long Jul 18 '16 at 12:32
• @RobertLong CI 5% was a typo. In the paper they present results as CI 95%, while SmartPLS3 actually returns two columns named 2.5% and 97.5%. Unfortunately I do not think I can post a reproducible example w/o uploading my model and data – nicolas Jul 18 '16 at 12:47
• The output makes no sense: if we understand "Orig. mean diff." to be the estimated value, then the reported CI of $[-0.056, 0.056]$ is too far away from $-0.264$ to be a plausible confidence interval. We have to wonder whether there's a bug, or you have misquoted something, or provide strange inputs to the procedure, or something else--but without a reproducible example, we cannot investigate these possibilities. – whuber Jul 18 '16 at 14:30
• @whuber Orig. mean diff is the difference of the mean values obtained for each of the original groups (see my edit with a longer explanation). SmartPLS also reports the mean of the mean-differences obtained in all permutations, which is 0.001. – nicolas Jul 18 '16 at 15:37
• Now that you have augmented the output, it appears to be telling us that the permutation mean difference likely likes between $-0.056$ and $0.056$. Since the actual mean difference is much further away from zero than that, you should conclude it is significant (and its p-value is $0.000$ to three decimal places). I'm having to guess because I am unfamiliar with your software. – whuber Jul 18 '16 at 16:02