What test should I use for a unipolar ordinal rating-scale? I am trying to analyze answers to 5-point unipolar rating items.
The possible answers can for instance be:
Not at all difficult, Slightly difficult, Moderately difficult, Very difficult, or Extremely difficult.
The respondents have answered each item twice, once regarding an assistive device they are currently using, and one regarding a proposed assistive device.
I'd like to answer:

*

*if any device is regarded better for each item;

*if a given item is
an issue for each device.

I'm assuming that the answers to the items are ordinal and can be asymmetrical. N = 14
For 1, I'm using a paired, two-sided Wilcoxon signed rank test, and am plotting the contingency matrix. I'm decently satisfied with this setup (but suggestions are always welcome).
It's mainly about 2 I'd like some feedback. All questions I find regarding this concerns two-sample tests and/or real likert items (bipolar items, e.g. agree/disagree). Since this is a one-sample test the Mann–Whitney U test many seem to rely on is ruled out, and as unipolar items probably are not symmetrical the Wilcoxon signed rank test shouldn't be used either.
The only nonparametric, one-sample test that does not assume symmetry I've found is the sign test. So currently I am using a one-sided sign test, testing that the median is not at 0 (Not at all difficult). In addition I am plotting histograms for the answers.
Does this sound right? Any feedback would be very appreciated!
 A: Regarding question 2: The first thing to acknowledge here is that it is your decision what counts as "an issue". It looks like the answer coded 0/zero ("not at all difficult") is the desirable response, and even "slightly difficult" is already something of a problem. On the other hand you do not expect in any case that everybody will respond zero, so it seems that you are willing to say that if there is a good proportion of zeroes but less than 100%, it's fine.
The problem here is that the data can't tell you which proportion of zeroes is fine. This has to be decided on subject matter grounds. You may be tempted to say that it's 50% on grounds that there is a statistical test that tests that, but (a) obviously this is not a reason to believe that that's the correct number on subject matter grounds, and (b) in principle you could test any other percentage based on the Binomial distribution (just test $q=0.8$ one-sided with $q$ being the probability for zero or $p=0.2$ as probability for nonzero by a Binomial test).
So you could well say "80% (or whatever) zeroes are required" and test that. Neither the data nor statistical theory can tell you which percentage to demand.
A problem with this kind of test (incl. your sign test idea) is that this will not take into account how the nonzeroes are distributed, i.e., 80% 0, 20% 1 is counted in the same way as 80% 0, 5% 1, 15% 4. It will be more difficult to construct tests if you want to set more complex requirements to take into account also the actual distribution of nonzeroes (for example you might want to say min 80% zero, max 5% 3 or 4). It may well be be possible but I don't know out of the top of my head how to do it (it will depend on what exactly your criterion is).
Another question is whether you need any test al all. You could also say that you decide whether there is an issue just from looking at the empirical numbers. If you did that, obviously there were no problem with a criterion such as "min 80% zero, max 5% 3 or 4", you could just check this directly.
I'm using the 80% criterion as example below but there's nothing special about 80%, ultimately you have to decide what your criterion is.
Advantage of using a test: You may want to avoid to declare an "issue" if what you see may just be caused by random variation. For example, if you say your criterion is min 80% zeroes and you see just 77%, you may think that in fact in the population there might be 80% and you just observed 77% because of an unlucky sample. (This would require thinking about whether you have something that can be treated as a proper random sample of a well defined population, because if you don't, a test will be invalid anyway.)
But keep in mind that if you observe 77%, these 77% are your best guess of how many zeroes you'd find in the population. If the test doesn't reject 80%, it doesn't mean that 80% is in fact true. The test gives you something of a "probability guarantee" (i.e., not a 100% reliable one;-) that if indeed the proportion is 80% or higher, your "alarm" will not normally be triggered, but the true proportion may well be lower. Whether this is a good thing depends on whether in a situation in which you observe below 80%, you'd be happy to not declare an issue because the true probability might be 80%, even though in fact it may also be lower. This depends on the consequences of this decision. It may also depend on the sample size, because if the sample size is reasonably big, you will not reject 80% only if the empirical (and also in all likelihood the true) population is pretty close to 80%, if potentially slightly lower, whereas if the sample size is low, you may still not reject 80% with a certain probability even in case of a true population proportion around 0.5, say. You may be willing to tolerate the former but not the latter.
So it all depends on (a) what your definition of a "borderline issue" is, and (b) what the consequences are of your "declaration of an issue" (or the absence of it).
