3
$\begingroup$

EDIT: More context in experiment, hypotheses, and more clarity in my questions. TL;DR below.

Fifty participants completed two similar tasks under three conditions:

  • No distraction
  • Low distraction
  • High distraction

A theory suggests both tasks measure the same mechanism. In addition, the theory says performance on both tasks will be highest in the no distraction condition, followed by the low distraction condition, with lowest performance in the high distraction condition.

I have already tested the following predictions on the continuous scores:

  • Performance on no dist. > lo dist. > hi dist. in Task 1 [TRUE]
  • Performance on no dist. > lo dist. > hi dist. in Task 2 [TRUE]
  • Performance on the two tasks will be correlated for each condition [FALSE]

Since the third prediction was incorrect using the continuous scores, I've decided to take an additional approach and reduce this data down to frequencies.

These are my observed values for 'best' scores:

Best Score           Best Score in Task 1
in Task 2      No Dist.      Lo Dist.      Hi Dist.
No Dist.          12            5             1    
Lo Dist.          14            8             2
Hi Dist.           4            3             1

And my observed values for 'worst' scores:

Worst Score           Worst Score in Task 1
in Task 2      No Dist.      Lo Dist.      Hi Dist.
No Dist.           1            2             4    
Lo Dist.           1            2             5
Hi Dist.           6            4            25

And mosaic plots for these values, where tile area relates to participant frequency:

Observed data

I'd like to test the hypotheses:

  • More participants will achieve their best scores in both tasks in the 'no distraction' condition, than in the 'low distraction' condition; (1,1) > (2,2)
  • Less participants will achieve their best scores in both tasks in the 'high distraction' condition, than in the 'low distraction' condition; (3,3) < (2,2)
  • More participants will achieve their worst scores in both tasks in the 'high distraction' condition, than in the 'low distraction' condition; (3,3) > (2,2)
  • Less participants will achieve their worst scores in both tasks in the 'no distraction' condition, than in the 'low distraction' condition; (1,1) < (2,2)

These hypotheses can be visualised with the mosaic plots below (n.b. arbitrary values).

Expected data

A classic chi-squared test of independence would not answer my specific hypotheses. In fact they suggest best/worst scores on the two tasks to be independent.

If the exact values from these plots are used, my expected values for best scores would be:

Best Score           Best Score in Task 1
in Task 2      No Dist.      Lo Dist.      Hi Dist.
No Dist.         12.5          8.3          4.2    
Lo Dist.          8.3          5.5          2.8
Hi Dist.          4.2          2.8          1.4

And for worst scores would be the inverse:

Worst Score           Worst Score in Task 1
in Task 2      No Dist.      Lo Dist.      Hi Dist.
No Dist.          1.4          2.8          4.2    
Lo Dist.          2.8          5.5          8.3
Hi Dist.          4.2          8.3         12.5

However, the magnitude of the frequencies in these plots are arbitrary. It's the relative proportions between conditions that are important. Therefore, testing these specific expected values may not be fair.

My questions: (TL;DR)

  1. Can I use specific expected values in a chi-squared test?
    • If so, would it be fair to use the arbitrary values in the latter two tables?
  2. Should I test against strong and weak versions of the hypothesis, in terms of expected values?
    • If so, could I compare their fits somehow? e.g. a likelihood ratio test
$\endgroup$
3
$\begingroup$

I would relax your null hypothesis a bit. I would say that there are people who switch condition, and when they do they will follow an independence model (what you test with a standard $\chi^2$-test), but that there is also a substantial portion of the respondents that remain on the diagonals. This is a classic hypothesis in log-linear analysis.

You test that by estimating two models: one that is only an independence model, and one that adds parameters for the diagonals. The exponentiated diagonal parameters are the factors by which the counts in the diagonal cells increase due to the people who choose to stay on the diagonal cells (or if they are smaller than 1, choose to avoid the diagonal cells): the expected number of people in the diagonal cell 1,1 increases by a factor 1.5 compared to an independence model due the people that per se want to remain in a diagonal cell. Then we compare the two models with a likelihood ratio test. In your data we canot reject the hypothesis that there is nothing special about the diagonal.

Here is how I would do this in Stata:

. // prepare the data
. clear

. input task1 task2 freq

         task1      task2       freq
  1. 1 1 12
  2. 1 2  5
  3. 1 3  1
  4. 2 1  4
  5. 2 2  3
  6. 2 3  1
  7. 3 1 14
  8. 3 2  8
  9. 3 3  2
 10. end

.
. gen diag = (task1==task2)*task1

.
. // look at the data
. tabdisp task1 task2, c(freq)

----------------------------
          |      task2
    task1 |    1     2     3
----------+-----------------
        1 |   12     5     1
        2 |    4     3     1
        3 |   14     8     2
----------------------------

. tabdisp task1 task2, c(diag)

----------------------------
          |      task2
    task1 |    1     2     3
----------+-----------------
        1 |    1     0     0
        2 |    0     2     0
        3 |    0     0     3
----------------------------

.
. // test whether diagonals are more common than
. // we would expect under an independence model
.
. // the independence model
. poisson freq i.task1 i.task2, irr

Iteration 0:   log likelihood = -14.980851
Iteration 1:   log likelihood =  -14.95665
Iteration 2:   log likelihood = -14.956607
Iteration 3:   log likelihood = -14.956607

Poisson regression                              Number of obs     =          9
                                                LR chi2(4)        =      31.07
                                                Prob > chi2       =     0.0000
Log likelihood = -14.956607                     Pseudo R2         =     0.5095

------------------------------------------------------------------------------
        freq |        IRR   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       task1 |
          2  |   .4444444   .1888526    -1.91   0.056     .1932531    1.022136
          3  |   1.333333   .4157397     0.92   0.356     .7236524    2.456674
             |
       task2 |
          2  |   .5333333   .1651038    -2.03   0.042     .2907319    .9783739
          3  |   .1333333   .0709721    -3.79   0.000     .0469734    .3784647
             |
       _cons |       10.8   2.834643     9.07   0.000     6.456713    18.06492
------------------------------------------------------------------------------

. est store independence

.
. // allow diagonals to be different
. poisson freq i.task1 i.task2 i.diag, irr

Iteration 0:   log likelihood = -14.724323
Iteration 1:   log likelihood = -14.672566
Iteration 2:   log likelihood = -14.672395
Iteration 3:   log likelihood = -14.672395

Poisson regression                              Number of obs     =          9
                                                LR chi2(7)        =      31.64
                                                Prob > chi2       =     0.0000
Log likelihood = -14.672395                     Pseudo R2         =     0.5188

------------------------------------------------------------------------------
        freq |        IRR   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       task1 |
          2  |   .5506669   .3871607    -0.85   0.396     .1388128    2.184481
          3  |   1.759949   .9537229     1.04   0.297     .6084581    5.090604
             |
       task2 |
          2  |   .6046409    .256736    -1.18   0.236      .263069    1.389714
          3  |   .1655643    .137213    -2.17   0.030     .0326231    .8402489
             |
        diag |
          1  |    1.54041   1.011098     0.66   0.510     .4255227    5.576352
          2  |   1.156618   .9645865     0.17   0.861      .225586    5.930179
          3  |   .8810864   .9542909    -0.12   0.907     .1054618    7.361081
             |
       _cons |   7.790132   4.592245     3.48   0.000     2.453384    24.73569
------------------------------------------------------------------------------

. est store diag

.
. // test the two models agains one another
. lrtest diag independence

Likelihood-ratio test                                 LR chi2(3)  =      0.57
(Assumption: independence nested in diag)             Prob > chi2 =    0.9036
$\endgroup$
  • $\begingroup$ Thanks for your very thorough answer. I haven't used Stata before and have never performed a likelihood ratio test, so your walkthrough is very helpful. I have some questions: 1) Why would we expect a ratio of 1.5? 2) What specific hypothesis does this test, and what would be the null? $\endgroup$ – E_Williams Jul 3 '17 at 12:13
  • $\begingroup$ 1) that is the empirical finding: In the last model you see the first exponentiated diag coefficient is 1.54041. In the text I interpreted that parameter. 2) The exponentiated diag coefficient are all 1 (the diag coefficients are all 0). $\endgroup$ – Maarten Buis Jul 3 '17 at 12:58
  • $\begingroup$ The explanation should now be clearer. $\endgroup$ – Maarten Buis Jul 3 '17 at 13:02
  • $\begingroup$ You don't need to use Stata for that. Any program that can estimate a poisson regression will do. I am just most comfortable with Stata. $\endgroup$ – Maarten Buis Jul 3 '17 at 13:04
  • $\begingroup$ Thanks again, Maarten. More Qs: a) Is the independence model the second table in my question? b) How would that hypothesis be worded in relation to my research question? You said I should relax my null, but I'm not sure what you mean. c) Do you have any links for using likelihood ratio tests / poisson regressions in this way? I spent yesterday googling but a lot of it didn't seem relevant. $\endgroup$ – E_Williams Jul 4 '17 at 11:18
1
$\begingroup$

You can test that hypothesis directly with a chi-square goodness of fit. Each cell of your design is a cell with a hypothesized proportion (this is what the test of independence is doing anyway, it just is coming up with a way of combining the marginals to come up with the appropriate proportions assuming independence).

Using raw counts...

$ \chi^2 = \Sigma\frac{(O - E)^2}{E} $

Where O is your observed count under the null and E is your expected count. Although it should be noted that E will fall apart under expectations of 0 (and generally chi-square doesn't perform well with low count cells... so some allowances may need to be made... a hypothesized proportion of 0 is a pretty extreme hypothesis).

$\endgroup$
  • $\begingroup$ Thanks RPierce! So I could come up with fairer expected values, and use a chi-square goodness of fit test? $\endgroup$ – E_Williams Aug 13 '17 at 13:02
  • $\begingroup$ So if the expected values are a good fit, chi-square would return a non-significant result? Is there an alpha criterion for poor fit? Perhaps I could combine your answer with Maarten's and do compare the fits of an independence model with a model with specified expected values. $\endgroup$ – E_Williams Aug 13 '17 at 13:23
  • $\begingroup$ The expected values are your null hypothesis, so deviation from expectation will lead to a higher chisq which in turn lead to a lower p value. In short, in a goodness of fit test a low p value reflects a statistically sig deviation from the tested hypothesis. Like any statistical test alpha is a matter of purpose, tast, and tradition. $\endgroup$ – russellpierce Aug 13 '17 at 14:39
  • $\begingroup$ When I was reading up I saw: "The chi-square goodness of fit test is applied when you have one categorical variable from a single population. It is used to determine whether sample data are consistent with a hypothesized distribution." Did you mean the chi-square test of independence? In which case, I'd like to compare the relative likelihoods of the null hypothesis, and a specific hypothesis. $\endgroup$ – E_Williams Aug 13 '17 at 14:49
  • $\begingroup$ That is an excessively restrictive description of the Goodness of Fit test IMO. I did mean Goodness of Fit (Independence is the same test just with the expected values parameterized by the marginal counts). $\endgroup$ – russellpierce Aug 14 '17 at 2:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.