2
$\begingroup$

As part of my research I have developed a software and I wanted to test it with 2 other softwares that does similar work. The problem is I could find only few users of these two softwares (20 users for software 1 and 10 users for software 2 and 5 users used both software 1 and software 2). This is what I have done

  1. Defined a 4 questions that covers different aspects of the software. These questions can be answered in a Likert scale of 1-5

  2. These 30 users are asked to answer these questions based on my software and other software (this 30 included users of software 1 or software 2. So I Considered software 1 and 2 as a single set)

  3. Done wilcoxon test to find the statistical significance

I want to know whether the procedure followed by me is good. Or is there any better procedure for my task at hand ? is it ok to consider software 1 and 2 as a single group to compare with my software ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Did the 5 users who are familiar with both software 1 and 2 form part of the 30 total? In other words, is the total number, $n$, of answers for the ratings $30$ or $35$? If there were $35$ binary comparisons, then you could use $35$ as the $n$ for comparison, with the result being a comparison of your software with "other" software, where "other" is weighted by frequency of use of "other."

Edit: OK, please confirm that you asked about each users experience with "any and all other software." In that case the comparison is paired between your software and the sum total experience for each user with other software, and, Wilcoxon signed-rank test is appropriate for $n=30$.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Yes. both software 1 and 2 form part of the 30 total. That is we have $30$ users in total for the study. My question is, is it ok to consider software 1 and 2 as a single group to compare with my software ? Another question is whether my testing procedure is ok ? $\endgroup$
    – user2721
    Commented Feb 28, 2018 at 0:52
  • $\begingroup$ From a research perspective is it ok to do a test like that to prove that my software is better than other softwares ? Or i need to do the test separately for each software ? $\endgroup$
    – user2721
    Commented Feb 28, 2018 at 4:24
  • $\begingroup$ Bit of an apples and oranges problem. The data you have does not have the same ability to discriminate between better and worse for each software comparison, yours with 1, yours with 2, and yours with 1 & 2. If you had equal numbers in each category you could contemplate running such a comparison with the same power, but, you do not have that. Moreover, each test would in that ideal case still have $\frac{n}{3}$ comparisons, not as powerful as with $n$ comparisons. What you do have is a comparison with software experience of users, not with any particular alternative software. $\endgroup$
    – Carl
    Commented Feb 28, 2018 at 4:40
  • $\begingroup$ Ok. I am going with my software vs Other. Which plot is best suited for this ? $\endgroup$
    – user2721
    Commented Feb 28, 2018 at 8:11
  • $\begingroup$ Wilcoxon result is a probability does not need plotting. What else is worth plotting depends on things I have not been shown. $\endgroup$
    – Carl
    Commented Feb 28, 2018 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.