1
$\begingroup$

I am interested in how people react to new information regarding how to solve maths puzzles. I give people a maths test scored out of 30, tell people new information, and give them a similar maths test, again out of 30. I want to find out whether the information has affected the difference in score in each participant.

Participants are sorted into one or two groups: one is given one set of information, the other a different set. So there are two treatments. However, I am also interested to see if an interaction between treatment and peoples university faculty(science or arts) will have any affect in how they tackle puzzles.

I would like to model my findings using a GLM. The obvious thing to do would be to take the difference in score before and after, and model the difference as a normal GLM. However, given the nature of the test, and the score being out of 30- the variance of the distribution of answers is most likely heterogenous- the variance at around a score of 15 will be larger than the variance around the score of 1 or 29.

One other option is to use the log-odds ratio, and calculate it for each participant.

$b_1=$ number of correct answers in before test $b_2=$ number of incorrect answers in before test $a_1=$ number of correct answers in after test $a_2=$ number of incorrect answers in after test $$ \text{Log Odds ratio} = \log(b_1/b_2)/(a_1/a_2) $$ This would solve my variance problem, but I am confused about how to model it, and if it is the same as logistic GLMs? Would I model the Log odds ratio using a normal GLM as I have negative values? How does this differ from a binomial GLM?

Would this allow me to model a mixed effects model using participant as a random effect using the log odds ratio?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It seems like your score is simply number of correct out of 30 questions. Then I would model that using a logistic regression, which have parameters that are log odds directly.

Your predictor variables would be a factor variable for person, a factor coding science/arts, and a factor for before/after, and interactions. Maybe the person factor should be a random effect (if using R you could use the package lme4). Then I would look out for possdible problems with overdispersion, which is always something to look out for with logistic regression. 

  EDIT

Below an example of logistic analysis with the example data you provided. First, the data file I constructed from the dropbox link:

# Block1C  number correct before
# Block1I  number incorrect before
# Block2C  number correct after
# Block2I  number incorrect after
# Treat    0=control, 1=Treat
# Fac      0=science, 1=arts   
Block1C Block1I Block2C Block2I Treat   Fac
20  10  17  13  1   1
16  14  21  9   1   1
16  14  13  17  1   1
20  10  21  9   1   1
16  14  18  12  1   1
10  20  12  18  1   1
15  15  11  19  1   1
17  13  12  18  0   0
17  3   21  9   0   0
19  21  14  16  0   0
18  12  22  8   0   0
17  13  19  11  0   0
18  12  11  19  0   1
14  16  13  17  0   1
7   23  1   29  0   1
18  12  15  15  0   1
18  12  17  13  0   1
16  14  13  17  0   1
12  18  13  17  0   0

Then, reading in and preparing (reshaping) the data for analysis. We use the melt function from package data.table, which really extends data.frame :

dat  <-  read.table(file="math_puzzles.txt",header=TRUE,
                    colClasses=c(rep("numeric",4),rep("factor",2)))
dat$Participant  <-  as.factor(1:19)
### Using data.table and melt():
library(data.table)   ### Extensions of data frames
dat  <-  as.data.table(dat)
dat2  <-  melt(dat, id=c("Participant","Fac","Treat"),
                         measure.vars=list(c("Block1C","Block2C"),
                                           c("Block1I","Block2I")),
               value.name=c("Correct","Incorrect"),
               variable.name="time", verbose=TRUE)

Then the logistic regression model with glm:

> mod1  <-  glm( cbind(Correct,Incorrect) ~ Fac+Treat+time,data=dat2,
+               family=quasibinomial)
> summary(mod1)

Call:
glm(formula = cbind(Correct, Incorrect) ~ Fac + Treat + time, 
    family = quasibinomial, data = dat2)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-5.0491  -0.9201   0.0306   1.2389   2.6465  

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept)   0.3056     0.1927   1.586    0.122  
Fac1         -0.4469     0.2371  -1.885    0.068 .
Treat1        0.3650     0.2280   1.601    0.119  
time2        -0.1418     0.1882  -0.754    0.456  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for quasibinomial family taken to be 2.495025)

    Null deviance: 104.615  on 37  degrees of freedom
Residual deviance:  92.964  on 34  degrees of freedom
AIC: NA

Number of Fisher Scoring iterations: 4

> confint(mod1)
Waiting for profiling to be done...
                  2.5 %     97.5 %
(Intercept) -0.07033639 0.68670110
Fac1        -0.91408500 0.01634683
Treat1      -0.08082370 0.81403122
time2       -0.51135618 0.22687954

Then finally a mixed effects logistic regression using the package lme4 :

> library(lme4)
> mod2  <-  glmer( cbind(Correct,Incorrect) ~ (1 | Participant)+ Fac+Treat+time,
+                 data=dat2, family=binomial)
> summary(mod2)
Generalized linear mixed model fit by maximum likelihood (Laplace
  Approximation) [glmerMod]
 Family: binomial  ( logit )
Formula: cbind(Correct, Incorrect) ~ (1 | Participant) + Fac + Treat +  
    time
   Data: dat2

     AIC      BIC   logLik deviance df.resid 
   216.9    225.1   -103.4    206.9       33 

Scaled residuals: 
     Min       1Q   Median       3Q      Max 
-2.28230 -0.50098  0.04307  0.46146  1.34688 

Random effects:
 Groups      Name        Variance Std.Dev.
 Participant (Intercept) 0.2199   0.469   
Number of obs: 38, groups:  Participant, 19

Fixed effects:
            Estimate Std. Error z value Pr(>|z|)  
(Intercept)   0.3561     0.2293   1.553   0.1203  
Fac1         -0.5145     0.3125  -1.646   0.0997 .
Treat1        0.3992     0.3005   1.328   0.1840  
time2        -0.1650     0.1222  -1.351   0.1769  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Correlation of Fixed Effects:
       (Intr) Fac1   Treat1
Fac1   -0.682              
Treat1  0.002 -0.521       
time2  -0.271  0.006 -0.004
> confint(mod2)   ### Takes some time ... 
Computing profile confidence intervals ...
                 2.5 %     97.5 %
.sig01       0.2954552 0.73555790
(Intercept) -0.1114361 0.83355966
Fac1        -1.1729185 0.12437142
Treat1      -0.2182774 1.02947320
time2       -0.4050066 0.07437656
Improve this answer
$\endgroup$
7
  • $\begingroup$ I have used a binomial model- The treatment has to nested with each participant when using random effects and then the model doesn't converge. $\endgroup$
    – mahyea
    Feb 17, 2017 at 12:12
  • $\begingroup$ How many subjects do you have? Why do you think you have to nest the treatment effect with participant when using random effects? If you think the effects differ by participant, maybe the treatment effect also could be a random effect? Then you would be interested in the mean of that random effect. Can you post your data? (Or, if not, simulated data with the same structure?) $\endgroup$
    – kjetil b halvorsen ♦
    Feb 17, 2017 at 12:16
  • $\begingroup$ I only have 48 participants which is why I also think log-odds is appropriate. How do I post a text file on here? Participant is the random effect, but the subject is either assign treatment 1 or 2, which affects the score of the participant. $\endgroup$
    – mahyea
    Feb 17, 2017 at 14:32
  • $\begingroup$ People often make a textfile available in dropbox and post the link here. If you are using R you can do dput(your_object_here) which we then can read into R using dget $\endgroup$
    – kjetil b halvorsen ♦
    Feb 17, 2017 at 14:43
  • $\begingroup$ dropbox.com/s/jwmpdvy0835105m/ex.txt?dl=0 $\endgroup$
    – mahyea
    Feb 17, 2017 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.