2
$\begingroup$

I've recently conducted an experiment where 24 participants performed a set of four tasks twice - using two different virtual reality interaction methods. The tasks consisted of a game similar to whac-a-mole, repeating hand gestures, pointing at targets and building a tower. Completing each task resulted in a score based on the performance of the participant. Half of the participants started with the first method (controllers) and the other half with the other one (hands). The scores have no inherent meaning outside of the task.

Here is score distribution for two of the tasks:

gesture task pointing task

I've tried reading about analyzing experiment data and I simply got overwhelmed with the number of methods in which it can be approached. I suspect I should use t-testing, and someone suggested I use Cohen's d for effect size, but honestly each of these methods has its own subvariants and I don't know which one (if any) I should choose.

What method is best for this type of experiment? Any examples or resources on how to use this method would be greatly appreciated.

Improve this question
$\endgroup$
4
  • $\begingroup$ What is the research question you want the test to address? $\endgroup$
    – dimitriy
    Commented Dec 2, 2022 at 1:35
  • $\begingroup$ Sorry, should have mentioned that. I want to find out whether there are performance differences between the interaction methods for each of the tasks. The tasks are designed to test different aspects of vr interactions so they can't be treated as one. $\endgroup$
    – Wojtek Wencel
    Commented Dec 2, 2022 at 2:47
  • $\begingroup$ Does order of exposure matter here or can you treat {controller, hands} users the same as {hands, controller} users? $\endgroup$
    – dimitriy
    Commented Dec 2, 2022 at 6:43
  • $\begingroup$ It matters in the way that participants learn the tasks (which are the same for hands and controllers) and could perform better on the second attempt because of this. To counteract this half if them have started with one method and the other half with the other method. $\endgroup$
    – Wojtek Wencel
    Commented Dec 2, 2022 at 9:35

1 Answer 1

Reset to default
1
$\begingroup$

It sounds like you want to test the null hypothesis that the score changes for all four tasks are zero for both treatment groups ({controller, hands} users and {hands, controller} users).

Here is an example where there are four simulated score difference variables (d1,d2,d3, and d4) and two treatment groups ({A, B} and {B, A}).

Here I use Seemingly Unrelated Regression (SUR) to predict the expected score change for each group and metric. I then perform a Wald test that all 4 tasks x 2 groups = 8 changes are the same, versus the alternative that they are not. With 24 observations, I asked Stata to return small-sample statistics. I also tried bootstrapping.

Here's the output:

. clear

. 
. /* (1) Generate Fake Data */
. set seed 12022022

. set obs 24
Number of observations (_N) was 0, now 24.

. generate id = _n

. generate treat_order = mod(_n,2)

. label define treat_order 0 "A, B" 1 "B, A"

. label values treat_order treat_order

. expand 4
(72 observations created)

. bysort id: gen j = _n

. generate d = rnormal(-1/4 + j + treat_order, 10)

. xtset id j

Panel variable: id (strongly balanced)
 Time variable: j, 1 to 4
         Delta: 1 unit

. reshape wide d, i(id treat_order) j(j)
(j = 1 2 3 4)

Data                               Long   ->   Wide
-----------------------------------------------------------------------------
Number of observations               96   ->   24          
Number of variables                   4   ->   6           
j variable (4 values)                 j   ->   (dropped)
xij variables:
                                      d   ->   d1 d2 ... d4
-----------------------------------------------------------------------------

. format %9.0fc d*

. list, noobs clean

    id   treat_~r    d1    d2    d3    d4  
     1        B,A    -5    11   -18   -10  
     2        A,B   -13   -12    -4    17  
     3        B,A     7     6    16    11  
     4        A,B   -16     5    -3    -4  
     5        B,A    12   -18     1   -10  
     6        A,B     1     0    -4   -18  
     7        B,A    -5     7   -13    14  
     8        A,B   -18   -10    -1     8  
     9        B,A    11    16     2   -10  
    10        A,B   -13    13     3    -5  
    11        B,A    -7    -4     7     2  
    12        A,B    -3    -7     7    16  
    13        B,A     2    13    14    26  
    14        A,B    16     9    -4     4  
    15        B,A     0     5     4    19  
    16        A,B   -17     5     0    16  
    17        B,A     0     1   -19    12  
    18        A,B    -3     8     5   -11  
    19        B,A    -2     2    -4    -1  
    20        A,B    -3    17     2   -10  
    21        B,A     4    -2     6    -5  
    22        A,B     7     8    11     6  
    23        B,A     7     3    12    -4  
    24        A,B    -8    -6    17     8  

. table treat_order, stat(mean d*)

---------------------------------------------------------
            |        1 d        2 d        3 d        4 d
------------+--------------------------------------------
treat_order |                                            
  A,B       |  -5.787531   2.383917   2.416916   2.126029
  B,A       |   2.008916   3.213855   .6086745   3.763657
  Total     |  -1.889307   2.798886   1.512795   2.944843
---------------------------------------------------------

. 
. /* (2a) SUR Model */
. sureg ///
>         (d1 ibn.treat_order, noconstant) ///
>         (d2 ibn.treat_order, noconstant) ///
>         (d3 ibn.treat_order, noconstant) ///
>         (d4 ibn.treat_order, noconstant) ///
> , small

Seemingly unrelated regression
------------------------------------------------------------------------------
Equation             Obs   Params         RMSE  "R-squared"         F      P>F
------------------------------------------------------------------------------
d1                    24        2     8.661508      0.2144       3.27   0.0425
d2                    24        2     9.107464      0.0953       1.26   0.2877
d3                    24        2     9.560491      0.0357       0.44   0.6424
d4                    24        2      11.9933      0.0662       0.85   0.4308
------------------------------------------------------------------------------

------------------------------------------------------------------------------
             | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
d1           |
 treat_order |
        A,B  |  -5.787531   2.393914    -2.42   0.018    -10.54493   -1.030129
        B,A  |   2.008916   2.393914     0.84   0.404    -2.748485    6.766318
-------------+----------------------------------------------------------------
d2           |
 treat_order |
        A,B  |   2.383917    2.51717     0.95   0.346     -2.61843    7.386264
        B,A  |   3.213855    2.51717     1.28   0.205    -1.788492    8.216201
-------------+----------------------------------------------------------------
d3           |
 treat_order |
        A,B  |   2.416916    2.64238     0.91   0.363    -2.834259    7.668091
        B,A  |   .6086745    2.64238     0.23   0.818    -4.642501     5.85985
-------------+----------------------------------------------------------------
d4           |
 treat_order |
        A,B  |   2.126029   3.314773     0.64   0.523    -4.461387    8.713444
        B,A  |   3.763657   3.314773     1.14   0.259    -2.823758    10.35107
------------------------------------------------------------------------------

. 
. /* (2b) Wald Test */
. test ///
>         (_b[d1:0.treat_order] = 0) ///
>         (_b[d1:1.treat_order] = 0) ///
>         (_b[d2:0.treat_order] = 0) ///
>         (_b[d2:1.treat_order] = 0) ///
>         (_b[d3:0.treat_order] = 0) ///
>         (_b[d3:1.treat_order] = 0) ///
>         (_b[d4:0.treat_order] = 0) ///
>         (_b[d4:1.treat_order] = 0)

 ( 1)  [d1]0bn.treat_order = 0
 ( 2)  [d1]1.treat_order = 0
 ( 3)  [d2]0bn.treat_order = 0
 ( 4)  [d2]1.treat_order = 0
 ( 5)  [d3]0bn.treat_order = 0
 ( 6)  [d3]1.treat_order = 0
 ( 7)  [d4]0bn.treat_order = 0
 ( 8)  [d4]1.treat_order = 0

       F(  8,    88) =    1.86
            Prob > F =    0.0772

. 
. /* (3a) BOOTSTRAPPED SUR Model */
. bs, reps(500) seed(12022022) nodots: sureg ///
>         (d1 ibn.treat_order, noconstant) ///
>         (d2 ibn.treat_order, noconstant) ///
>         (d3 ibn.treat_order, noconstant) ///
>         (d4 ibn.treat_order, noconstant) ///
> , small

Seemingly unrelated regression
------------------------------------------------------------------------------
Equation             Obs   Params         RMSE  "R-squared"         F      P>F
------------------------------------------------------------------------------
d1                    24        2     8.661508      0.2144       3.27   0.0425
d2                    24        2     9.107464      0.0953       1.26   0.2877
d3                    24        2     9.560491      0.0357       0.44   0.6424
d4                    24        2      11.9933      0.0662       0.85   0.4308
------------------------------------------------------------------------------

------------------------------------------------------------------------------
             |   Observed   Bootstrap                         Normal-based
             | coefficient  std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
d1           |
 treat_order |
        A,B  |  -5.787531   2.830594    -2.04   0.041    -11.33539   -.2396685
        B,A  |   2.008916   1.691195     1.19   0.235    -1.305765    5.323598
-------------+----------------------------------------------------------------
d2           |
 treat_order |
        A,B  |   2.383917   2.541327     0.94   0.348    -2.596993    7.364827
        B,A  |   3.213855   2.473485     1.30   0.194    -1.634088    8.061797
-------------+----------------------------------------------------------------
d3           |
 treat_order |
        A,B  |   2.416916   1.809315     1.34   0.182    -1.129276    5.963108
        B,A  |   .6086745   3.349253     0.18   0.856    -5.955741     7.17309
-------------+----------------------------------------------------------------
d4           |
 treat_order |
        A,B  |   2.126029   3.266604     0.65   0.515    -4.276398    8.528455
        B,A  |   3.763657   3.676691     1.02   0.306    -3.442525    10.96984
------------------------------------------------------------------------------

. 
. /* (3b) BS Wald Test */
. test ///
>         (_b[d1:0.treat_order] = 0) ///
>         (_b[d1:1.treat_order] = 0) ///
>         (_b[d2:0.treat_order] = 0) ///
>         (_b[d2:1.treat_order] = 0) ///
>         (_b[d3:0.treat_order] = 0) ///
>         (_b[d3:1.treat_order] = 0) ///
>         (_b[d4:0.treat_order] = 0) ///
>         (_b[d4:1.treat_order] = 0)

 ( 1)  [d1]0bn.treat_order = 0
 ( 2)  [d1]1.treat_order = 0
 ( 3)  [d2]0bn.treat_order = 0
 ( 4)  [d2]1.treat_order = 0
 ( 5)  [d3]0bn.treat_order = 0
 ( 6)  [d3]1.treat_order = 0
 ( 7)  [d4]0bn.treat_order = 0
 ( 8)  [d4]1.treat_order = 0

           chi2(  8) =   15.97
         Prob > chi2 =    0.0428

I think I would use the bootstrapped version.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.