Appropriate Analysis for ordinal variable, repeated 4 times under different conditions, by the same 2 raters I am doubting myself on which analysis to run for the following:
18 participants were evaluated at 4 time points with different conditions at each time.
They were given scores (on a discrete visual analog scale) by 2 raters.
The scores were calculated for a pair of participants: the pairs changed at each time point.
I do know which participant comprises each pair.
Is that a 2-way repeated measures ANOVA? Some variation of Friedman test?
 A: Try to understanding your data before you do an ANOVA. First, plot your data to ensure that it is normally distributed.  Then plot total score by rater to determine if there is a difference between raters.  Estimate mean and 95% CI for the mean for each rater. Then estimate mean and 95% CI for the mean for the average total score of the raters for each of the four conditions.  Check and see conditions don't have overlapping of the confidence intervals. Try doing boxplots with notches to visualize overlap. ANOVA will tell you if one of the conditions is different, but not which.
A: If each rater gave 9 ratings -- one for each pair of subjects -- at each time point, if the subjects were re-paired at each time point, and if it is not known who was paired with whom, then I don't think there is any proper way to analyze the data, because of the unknown and unestimatable correlations among the ratings from from one time to the next.
If you are willing to treat the data as if there were 18 different subjects at each time point (72 subjects total) then you could do a 1-between, 1-within analysis, where "between" and "within" are relative to the pairs, which would play the role that is usually played by subjects in such analyses: pairs are nested within time and crossed with raters; time is the between-pairs (grouping) factor -- to check for a time difference, you must look at different pairs -- and raters is the within-pair (repeated measures) factor -- you can check for a rater difference within each pair.
