1
$\begingroup$

I am having difficulty determining a justifiable region of practical equivalence (ROPE) for a parameter from a multilevel probit model

Below is the posterior distribution for the fixed-effect of condition (left) and the intercept (right)

Figure 1

The study is essentially testing the feasibility of a new technology (does it show promise and/or does it warrant further research).

each condition refers to a learning tool being used. condition 1 is the established method (costly and difficult for many to access). condition 2 reflects a new method being tested (much more accessible). I'm most interested in assessing the noninferiority of condition 2 compared to 1.

the DV is accuracy (binary) and each participant had 4 data points per condition. average accuracy was 78% for condition 1 and 75% for condition 2. (not a lot of data, but experiment required a difficult to recruit population and was rather lengthy).

should the lower limit of the ROPE be selected based on the design/collected data? Based on the early stage of development, a performance difference between the two conditions of 1 out of 4 correct is acceptable. However, this would also reflect a 25% difference in accuracy.

If this was justified, would I then define the boundary based on the mean posterior density for the accuracy of condition 1 (here it is the intercept: .922 or 82% accuracy)? that would set the lower bound (for the difference) as -0.741 (see below).

Figure 2

Questions:

  • Should I be considering equivalence/noninferiority based on values that could be obtained in the experiment as in the example above?
  • Should I be using some other measure?

additional info/question

  • 95% credible interval for condition 1 (52.1% - 97.1%) accuracy
  • can I take the HDI for fixed effect (probit: -.747,.78) and then adjust the mean of posterior distribution for the intercept (probit: .922) to determine the the 95% credible interval for condition 2? the result would be: (56.9% - 95.6%) accuracy.

Edit: added the model description below

model

a and b are random effects (varying intercept) for subjects and items/targets. alpha is the fixed effect for condition.

$\endgroup$
  • 1
    $\begingroup$ Just to make sure I understand this properly... You want to evaluate some new treatment based around accuracy on a test. You have 4 tests with business as usual and 4 tests with the treatment. You are hoping to make a statement along the lines of "The impact of this new treatment does not hurt accuracy by more than 25%" and you are are looking for a ROPE to do this. $\endgroup$ – Kitter Catter May 16 at 20:34
  • $\begingroup$ That is correct $\endgroup$ – bag2 May 16 at 20:35
  • 1
    $\begingroup$ I think I'm not understanding your procedure entirely, but why wouldn't you just use the difference in accuracy between 2 and 1 must be greater than -0.25 from your first graph then? Are you using some nonlinearities in a link function? It might help if you write out your model in the question so we have a clearer idea on your approach. I'm kinda simple so having those equations are really helpful for me. $\endgroup$ – Kitter Catter May 16 at 22:04
  • $\begingroup$ I am using a probit link function. Part of what I am unsure of is if I should use a probit coefficient that corresponds to -25% (-0.675), or a coefficient that corresponds to 25% reduced performance when compared to estimated performance for condition 1 (.922 or 82% accuracy). reducing 82% by 25%, results in 57% accuracy which would then correspond to reducing the intercept by .741 (a probit coefficient of 0.181). I'll update the main post with the model. $\endgroup$ – bag2 May 16 at 22:17
  • $\begingroup$ ultimately, the these data are highly unlikely support a conclusion of noninferiority (need a larger sample size); however, I want to ensure the final value I compare against is based on sound reasoning $\endgroup$ – bag2 May 16 at 22:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.