How to compare outcomes from single variable experiment?

Question

I've performed a study which yielded (?) the following results:

-                   no bike box      bike box     % change
correct procedure      173              55           -27%
incorrect procedure    68               50           69%

Since a result could only be one of the two - correct and incorrect procedure, do I need both quantities in my data, or should I just interpret one of them? If so, which one should I use, or does it depend what i'm trying to demonstrate?

Sorry for being so vague, but I hope this is enough to answer my question.

Answer 1

You can report percent correct along with sample size $n$, and reporting percent correct instead would be sufficient in most cases, even if you focus more on the percent incorrect in your interpretation.

Answer 2

If I understand correctly, your IV is "bike box" vs. "no bike box", and your DV is "correct" vs. "incorrrect". The resulting $2 \times 2$ classification table can be summarized with the Odds Ratio: "given the bike-box condition, what are the odds of getting a correct response?" compared to "given the no-bike-box condition, what are the odds of getting a correct response?" If the odds are identical, OR is 1. Yule's Q standardizes OR to the $[-1, 1]$ interval. In R:

> IV    <- factor(rep(c("no bbox", "bbox"), c(241, 105)))
> DVnbb <- rep(c("correct", "incorrect"),   c(173,  68))
> DVbb  <- rep(c("correct", "incorrect"),   c( 55,  50))
> DV    <- factor(c(DVnbb, DVbb))
> cTab  <- table(IV, DV)
> addmargins(cTab)
         DV
IV      correct incorrect Sum
bbox         55        50 105
no bbox     173        68 241
Sum         228       118 346

> library(vcd)                   # for oddsratio()
> (OR <- oddsratio(cTab, log=FALSE))
[1] 0.4323699

> (55/50) / (173/68)             # check: ratio of odds
[1] 0.4323699

> (Q <- (OR-1) / (OR+1))         # Yule's Q
[1] -0.3962873

A corresponding test for equal distributions of your DV within IV groups is Fisher's test.

> fisher.test(cTab)
Fisher's Exact Test for Count Data
data:  cTab 
p-value = 0.0008111
alternative hypothesis: true odds ratio is not equal to 1 
95 percent confidence interval:
0.2619504 0.7158848 
sample estimates:
odds ratio 
0.4334897

Note that fisher.test() does not report the empirical OR, but a maximum-likelihood estimation.

Edit: Reading your answer, another measure that might capture some relevant information is relative risk: its definition is very similar to OR but calculates the "risk" of getting a correct response given one of the two conditions (and not the odds), i.e., the conditional relative frequency of a correct response.

# risk of getting correct (1st column) response in the two conditions
# calculated as conditional frequency: (cell count) / (sum of row counts)
> (risk <- prop.table(cTab, margin=1))
              DV
IV        correct incorrect
bbox    0.5238095 0.4761905
no bbox 0.7178423 0.2821577

# compare risk in experimental condition to risk in control condition
> (relRisk <- risk[1, 1] / risk[2, 1])
0.7297

Answer 3

my brother posted the original data.

what is happening here is that i am observing two different conditions. the control is no bike box. this is the baseline. a normal intersection with bike lanes. the experimental condition is an intersection that has bike boxes added. i am attempting to interpret the correct/incorrect use as a function of this change in the environment. basically, i want to know how best to report the data in a simple and understandable way.

the problem i find it that stuff gets messed up because boxes/no boxes are independent from the correct/incorrect behaviour. i think.

if you take this as percentages

           correct       incorrect

no box (control) 72% 28%

bbox (exp) 52% 48%

i basically want to have a descriptive value so i can explain how the bike boxes effect the outcome of correct or incorrect behaviour at a specific intersection. the two intersections have different sample sizes which complicate things from my basic psych stats text and knowledge. what i am also thinking is that i need to use the no box data as my baseline in which to compare the experimental condition? that makes sense, no?

what i was thinking is that i could report a decrease of 20% correct behaviour and an increase of 20% for incorrect responses. so does that mean there is a total difference of 40%? or should i just focus on the 20% and report for one variable, correct or incorrect as to which i see as more important for the report. but to me this data seems incomplete. it doesnt really tell the whole story.

i also thought about doing a percent change ie (new val/old val)-1. but again, i used the percentages of the total per outcome (correct incorrect) as reported above. so i then got those values:

-27% change for correct behaviour due to addition of bike boxes +69% change for incorrect behaviour due to addition of bike boxes

does that make any sense to what i am trying to show??

i am basically confused as to what makes the best sense to report in this case given the comparison i am making and what is the most accurate data that describes these changes.

thank you all.