Repeatedly measured, longitudinal and ordinal data

Question

Description of the problem: The effect of a certain treatment is tested as follows: For n subjects, the endpoint is measured three times before and after the treatment is administered, respectively. The endpoint is modeled as an ordinal variable with outcomes A,B,C and order $A<B<C$.

There are two questions to be answered: 1) How precise is the method measuring the endpoint? 2) Is the treatment effective?

Statistical description of the problem: We consider a random variable $Y_{itk}$ with $i=1,\ldots,n$ the subject index, $t=1,2$ the time point index, and $k=1,2,3$ the index of the repeated measurements. The random variable $Y_{itk}$ has the three outcome possibilities A,B,C which can be ordered as mentioned above. Also, the random variables are independent in $i$ but, of course, neither in $t$ nor in $k$.

Regarding the two questions:

1) What would be a good precision measure? I thought about the probability that a) all three results are the same, b) two are the same, c) all three measurements are different. For example the first probability could be estimated as follows: $\frac{1}{2n}\sum_{i,t}\mathbf{1}_{\{Y_{it1}=Y_{it2}=Y_{it3}\}}$. However, do I need to adjust of the dependence of the measurements at times $t=1$ and $t=2$? What other precision measure could be considered?

2) Concerning the comparison of the endpoint between the two time points, I have several ideas.

2.1) Defining a ranking of the outcomes of $(Y_{it1},Y_{it2},Y_{it3})$ and comparing the the distribution of the ranks for both time points. The things that bothers me is that the ranking system is up to a certain degree arbitrary and the results might depend on the ranking system.

2.2) Taking the median of the outcomes $\{Y_{it1},Y_{it2},Y_{it3}\}$. In the end, one could compare the distributions of the medians at times $t=1$ and $t=2$.

2.3) Multinomial regression with a random predictor. Eventually, the sample size $(n=10)$ might be to small for a fancy model.

Any advice is highly appreciated. Thanks!

By the way, a similar question has already been asked here. An answer hasn't been given though.

Answer 1

1 is a question of interrater agreement. Been studied extensively. Cohen's Kappa is often used for clusters of 2 observations, with ratings classified as either % +concordant, % -concordant, and % discordant. It accounts for the estimated prevalence of a binary condition, unlike % agreement. If evaluations are polytomous, you can use weighted kappas. However, given the sophistication, I would evaluate a simple % agreement because it's intuitively much easier for a reader to standardize the chances of such agreement given the prevalence of conditions than to interpret complex statistical measures without that information in context. I would look at times all 3 ratings agree entirely versus times in which any 1 rating possibly differs by any amount.

2 Personally, I would use a bootstrap approach to estimating associations with the $Y_{i,t}$s . I would randomly sample 1 pre-test per cluster and 1 post-test per cluster and iterate this, combining standard error estimates using Rubin's Rules. It's a bit sophisticated, but you can think of the results as not being conditional upon the value of the test, but conditional upon the presence or absence of conditions (incorporating uncertainty due to poor assessment).