5
$\begingroup$

I've ran an experiment in which different situations were presented to subjects in a random order, on which I measured a result out of $y \in {-1,0,1}$.

I'd like to show that $y$ is order-independent, meaning that the answer ratio was the same either if it was on the 1st trial or the last one.

My data is clearly shows it here:

data results

What would be the right statistical test to prove this claim?

ANOVA feels almost there, since the groups (the number of trials) are linked.

Moreover, if the imageorder was an arbitrary (and not ordered), ANOVA would still unfit since the in-group distribution doesn't fit anova's assumptions, and therefore we need a non-parametric test.

Any suggestions?

|cite|improve this question
$\endgroup$
  • $\begingroup$ A few questions: Do you expect that the influence of position-in-the-order (PIO) should be monotonic if H1 is rejected, or do you need to test for idiosyncratic/non-monotonic effects of PIO? Is a response of 0 somehow "between" 1 and -1, or is the response purely categorical? Do you expect the influence of PIO on response, if it exists, will be similar across all of the (presumably 13) situations? $\endgroup$ – Jacob Socolar Sep 5 '17 at 4:32
  • $\begingroup$ I'm not sure that monotonic test (linear effect if I understand you correctly) is the only thing I should test. The claim that I'm trying to prove is - no matter when the situation is presented, the answer distribution should be the same. $\endgroup$ – Dimgold Sep 6 '17 at 7:30
  • $\begingroup$ Do the subjects repeat, in the sense of each subject seeing multiple situations each presented in random order, or does each subject only contribute one response to the overall experiment? $\endgroup$ – jbowman Sep 7 '17 at 18:10
  • $\begingroup$ Yes, every subject (out of about 40) is observing 12 situations in a random order. $\endgroup$ – Dimgold Sep 8 '17 at 6:42
  • $\begingroup$ I think some kind of permutation test would fit this situation. However, you have a bigger problem. The null hypothesis is that order does not matter. You are trying to confirm the null hypothesis. From the look of your data, you have very unbalanced classes of outcomes: almost all data points have $Y = -1$. This will reduce the power of your test. If you have a low-powered test, then "failing to reject the null hypothesis" does not mean much. $\endgroup$ – Lizzie Silver Sep 9 '17 at 10:06
1
$\begingroup$

As I understand Your problem is, that You could not use the usual ANOVA for the problem. As the in-group distribution doesn't fit anova's assumptions, and therefore You need a non-parametric test. Idea of the such a test is well described here and here. My implementation of numeric non-parametric ANOVA is here.

Links to code

ANOVA function

Example

fPlotHist function

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.