2
$\begingroup$

I conducted a usability test of two password systems ps1 and ps2. Same users participated in the usability of both system. Analysis of average login attempts and average login time clearly indicates that ps1 is more usable than ps2. However, I would like to know which statistical tests are more relevant for comparing the usability of ps1 vs ps2. Or are mean login attempts and login time sufficient indicators?

Also I read about t-test which is used to compare two different samples. However in my study I have used the same sample (users) to determine the usability of ps1 and ps2.

Improve this question
$\endgroup$
4
  • $\begingroup$ You might find useful reading about a/b testing. This might be an interesting piece to get the basics of design of experiments. $\endgroup$
    – IcannotFixThis
    Commented Aug 3, 2015 at 10:53
  • $\begingroup$ Thanks, but a/b testing assumes testing two variants on different samples (users). I have tested two variants on the same samples (users). $\endgroup$
    – Curious
    Commented Aug 3, 2015 at 11:48
  • $\begingroup$ Can I use a Mann-Whitney U test? $\endgroup$
    – Curious
    Commented Aug 3, 2015 at 11:59
  • $\begingroup$ Mann-Whitney U test still comes with strong assumptions about independence. Can those hold? How did you execute your experiment? $\endgroup$
    – IcannotFixThis
    Commented Aug 3, 2015 at 13:29

1 Answer 1

Reset to default
1
$\begingroup$

Whether just reporting the means (presumably with confidence intervals) is sufficient is a judgment about your intended audience. We couldn't answer that for you.

If you want to submit these analyses to a peer-reviewed scientific journal that required a test and a p-value, you should not use t-tests. Neither of your measures could really be normally distributed. Your number of login attempts is a kind of count variable (specifically a negative binomial with $r=1$, cf, here), and the login time is a survival time. There are appropriate models for both. Also bear in mind that you need to account for the nonindependence in your data. Probably the simplest option for you would be to use the Wilcoxon signed rank test. My guess is that these contain the same information, and you could get away with just testing one of them.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.