1
$\begingroup$

I am trying to analyze answers to 5-point unipolar rating items.

The possible answers can for instance be: Not at all difficult, Slightly difficult, Moderately difficult, Very difficult, or Extremely difficult.

The respondents have answered each item twice, once regarding an assistive device they are currently using, and one regarding a proposed assistive device.

I'd like to answer:

  1. if any device is regarded better for each item;
  2. if a given item is an issue for each device.

I'm assuming that the answers to the items are ordinal and can be asymmetrical. N = 14

For 1, I'm using a paired, two-sided Wilcoxon signed rank test, and am plotting the contingency matrix. I'm decently satisfied with this setup (but suggestions are always welcome).

It's mainly about 2 I'd like some feedback. All questions I find regarding this concerns two-sample tests and/or real likert items (bipolar items, e.g. agree/disagree). Since this is a one-sample test the Mann–Whitney U test many seem to rely on is ruled out, and as unipolar items probably are not symmetrical the Wilcoxon signed rank test shouldn't be used either.

The only nonparametric, one-sample test that does not assume symmetry I've found is the sign test. So currently I am using a one-sided sign test, testing that the median is not at 0 (Not at all difficult). In addition I am plotting histograms for the answers.

Does this sound right? Any feedback would be very appreciated!

$\endgroup$
6
  • 1
    $\begingroup$ What does "a given item is an issue for each device" exactly mean? It seems you optimally want all answers at 0 (not at all difficult), but are prepared to tolerate some deviation. A test will not tell you how much deviation you should be willing to tolerate. You can test median=0 with the sign test, alright, but whether median=0 is actually what you're interested in, neither we nor the data can know. $\endgroup$ Jul 13, 2022 at 16:54
  • $\begingroup$ @ChristianHennig Well put! It means basically what you're suggesting: if an item, say "Using the device is mentally..." has a median=0 ("not at all taxing"), I would interpret that as that the device in question doesn't have an issue with this. It feels a bit arbitrary, but would it then make sense to for instance test the alternate hypothesis median < 0.5? $\endgroup$
    – JohanPI
    Jul 13, 2022 at 18:51
  • $\begingroup$ If you reject, there's evidence that the median is in fact smaller than 0.5. It really boils down to your decision. If this is what you think an "issue" means, then this is OK (of course you also have to decide the test level). I'm not really sure how you think we know better than you whether this makes sense, as obviously you understand the setup and we don't. $\endgroup$ Jul 13, 2022 at 23:23
  • 1
    $\begingroup$ To be somewhat provocative: I have the impression that you are not at all sure whether median=0 is what you think an issue really means, but want to test this because you have found a test that does it. If you had a different definition, it may be possible to construct a tailor-made test. It is not clear whether you need a statistical test at all. You could also say, more than 20% nonzero is too much, absolute, in the numbers, without testing. Your decision. Why do you think you need a test? $\endgroup$ Jul 13, 2022 at 23:27
  • $\begingroup$ @ChristianHennig Thank you for your comments, this is exactly the kind of feedback I was hoping for! $\endgroup$
    – JohanPI
    Jul 14, 2022 at 6:41

1 Answer 1

1
$\begingroup$

Regarding question 2: The first thing to acknowledge here is that it is your decision what counts as "an issue". It looks like the answer coded 0/zero ("not at all difficult") is the desirable response, and even "slightly difficult" is already something of a problem. On the other hand you do not expect in any case that everybody will respond zero, so it seems that you are willing to say that if there is a good proportion of zeroes but less than 100%, it's fine.

The problem here is that the data can't tell you which proportion of zeroes is fine. This has to be decided on subject matter grounds. You may be tempted to say that it's 50% on grounds that there is a statistical test that tests that, but (a) obviously this is not a reason to believe that that's the correct number on subject matter grounds, and (b) in principle you could test any other percentage based on the Binomial distribution (just test $q=0.8$ one-sided with $q$ being the probability for zero or $p=0.2$ as probability for nonzero by a Binomial test).

So you could well say "80% (or whatever) zeroes are required" and test that. Neither the data nor statistical theory can tell you which percentage to demand.

A problem with this kind of test (incl. your sign test idea) is that this will not take into account how the nonzeroes are distributed, i.e., 80% 0, 20% 1 is counted in the same way as 80% 0, 5% 1, 15% 4. It will be more difficult to construct tests if you want to set more complex requirements to take into account also the actual distribution of nonzeroes (for example you might want to say min 80% zero, max 5% 3 or 4). It may well be be possible but I don't know out of the top of my head how to do it (it will depend on what exactly your criterion is).

Another question is whether you need any test al all. You could also say that you decide whether there is an issue just from looking at the empirical numbers. If you did that, obviously there were no problem with a criterion such as "min 80% zero, max 5% 3 or 4", you could just check this directly.

I'm using the 80% criterion as example below but there's nothing special about 80%, ultimately you have to decide what your criterion is.

Advantage of using a test: You may want to avoid to declare an "issue" if what you see may just be caused by random variation. For example, if you say your criterion is min 80% zeroes and you see just 77%, you may think that in fact in the population there might be 80% and you just observed 77% because of an unlucky sample. (This would require thinking about whether you have something that can be treated as a proper random sample of a well defined population, because if you don't, a test will be invalid anyway.)

But keep in mind that if you observe 77%, these 77% are your best guess of how many zeroes you'd find in the population. If the test doesn't reject 80%, it doesn't mean that 80% is in fact true. The test gives you something of a "probability guarantee" (i.e., not a 100% reliable one;-) that if indeed the proportion is 80% or higher, your "alarm" will not normally be triggered, but the true proportion may well be lower. Whether this is a good thing depends on whether in a situation in which you observe below 80%, you'd be happy to not declare an issue because the true probability might be 80%, even though in fact it may also be lower. This depends on the consequences of this decision. It may also depend on the sample size, because if the sample size is reasonably big, you will not reject 80% only if the empirical (and also in all likelihood the true) population is pretty close to 80%, if potentially slightly lower, whereas if the sample size is low, you may still not reject 80% with a certain probability even in case of a true population proportion around 0.5, say. You may be willing to tolerate the former but not the latter.

So it all depends on (a) what your definition of a "borderline issue" is, and (b) what the consequences are of your "declaration of an issue" (or the absence of it).

$\endgroup$
1
  • $\begingroup$ Thank you for your comprehensive and thoughtful answer! As I noted earlier, this is the sort of feedback I was hoping for, and you've given me a lot to consider :) $\endgroup$
    – JohanPI
    Jul 14, 2022 at 12:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.