2
$\begingroup$

I am new to this website, and I hope to find help. I am writing my MSc thesis on phishing education.

I have three groups and use pretest-posttest experimental design

  1. Control
  2. Training 1
  3. Training 2

It aims to test whether anti-phishing training is effective in educating users to recognise phishing emails. I also use few control variables such as demographics ones, IT self-efficacy, etc.

The dependent measure is the final score.

These are the tests that I used:

  • One-way Anova followed by Post hoc Tukey and paired t-test to compare the scores between pre and post-test. However, it turned out that my post-test is more difficult than my pre-test; this is a big limitation (I don't know how to address this issue)
  • Signal detection theory: One-way Anova followed by Post hoc Tukey and paired t-test was used to examine: False negatives and false positives rates/Sensitivity and criterion.

I would like to know whether my tests are sufficient or I need additional analysis?

$\endgroup$
1
  • $\begingroup$ What’s the outcome? Score of what? $\endgroup$ Aug 12, 2021 at 3:51

1 Answer 1

2
$\begingroup$

Without additional information, all I can give is general advice.

If your design involved measuring some sort of ability to detect phish attempts prior to and after an intervention, then the best approach is to use an ANCOVA.

Model the post test score as a function of the pre test score and the exposure group. Mathematically

$$ y_{post} = \beta_0 + \beta_1y_{pre} + \beta_2 x_{T_1} + \beta_3 x_{T_2} $$

Why this way? A few reasons listed here by Frank Harrell, and here by Bland & Altman. In short, you risk a ton of bias if you do pre-post. Bland & Altman say it quite plainly in that article

Whether the outcome is change score or post score, one should always adjust for baseline using analysis of covariance (ANCOVA); otherwise, the estimated treat effect may be biased.

If you want to know which intervention is "best" you can use a Tukey's HSD but honestly the type one error rate is not inflated all that much. You could probably just use a smaller $\alpha$ in your initial model and end up with a 5% type one error rate after the multiple comparisons.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.