I am having difficulty determining a justifiable region of practical equivalence (ROPE) for a parameter from a multilevel probit model
Below is the posterior distribution for the fixed-effect of condition (left) and the intercept (right)
The study is essentially testing the feasibility of a new technology (does it show promise and/or does it warrant further research).
each condition refers to a learning tool being used. condition 1 is the established method (costly and difficult for many to access). condition 2 reflects a new method being tested (much more accessible). I'm most interested in assessing the noninferiority of condition 2 compared to 1.
the DV is accuracy (binary) and each participant had 4 data points per condition. average accuracy was 78% for condition 1 and 75% for condition 2. (not a lot of data, but experiment required a difficult to recruit population and was rather lengthy).
should the lower limit of the ROPE be selected based on the design/collected data? Based on the early stage of development, a performance difference between the two conditions of 1 out of 4 correct is acceptable. However, this would also reflect a 25% difference in accuracy.
If this was justified, would I then define the boundary based on the mean posterior density for the accuracy of condition 1 (here it is the intercept: .922 or 82% accuracy)? that would set the lower bound (for the difference) as -0.741 (see below).
- Should I be considering equivalence/noninferiority based on values that could be obtained in the experiment as in the example above?
- Should I be using some other measure?
- 95% credible interval for condition 1 (52.1% - 97.1%) accuracy
- can I take the HDI for fixed effect (probit: -.747,.78) and then adjust the mean of posterior distribution for the intercept (probit: .922) to determine the the 95% credible interval for condition 2? the result would be: (56.9% - 95.6%) accuracy.
Edit: added the model description below
a and b are random effects (varying intercept) for subjects and items/targets. alpha is the fixed effect for condition.