Before-after analysis of categorical data

Question

I have carried out an experiment, during which the participants tried several authentication methods; as well as a pre- and post-survey, which measured the participants' familiarity and adoption of said methods, their opinion of them, as well as motivation to use them. The questionnaires both have the same questions (e.g. On a scale from 1 to 5, please provide your opinion on monetary cost, associated with the use of each authentication method), and I'm using Paired T-Test on the means that I calculate from the answers to these 5-point Likert scale questions.

However, there are a couple of questions that have categorical answers (for example, for Do you use the following authentication methods? I could have simple Yes/No answers, and for How often do you use them? I could have a few categorical choices Several times a day, Once a day, Several times a week,..., or another simple Yes/No for Have you considered changing your current method of authentication for service X?).

I would still like to find out, whether the participants' opinions and practices have changed in between the first and second survey, when the experiment was conducted. Is there any test I could use to do before-after analysis on such categorical data, or is my only option to describe it using Descriptive statistics (e.g. Pre - Yes: 60%, No: 40% and Post - 80%, No: 20%)?

While I'm at it, I would also like to ask, whether it's possible to observe the relation between the participants' opinion from the survey (Q: Provide your opinion on time required to login, 5-point Likert scale ranging from Very short to Very long), and actual experimental data (Login time measured in seconds per participant).

Answer 1

You can use McNemar's test to do before-after analysis on categorical data, provided if you have only 2x2 contingency table. if it is not the case you dont have another test to perform before and after analysis as from my learning i'm just putting forward it.

Answer 2

You appear to have two types of data: ordinal and binary. It doesn’t matter whether you give them 1 to 5 or a Likert scale...the responses is just picking a single answer from somewhere in that order.

A quick discussion of what to use on Likert scales (and other ordinal scales) can be found here.

Notice that you must check your assumptions before attempting a t-test because there is a fair chance this type of data will violate assumption (such as normality) of a paired t-test. A non-parametric alternative is Mann-Whitney. However, there is no free lunch. If you use a non-parametric you also lose power and it is harder to identify small differences. But it has less requirements before you use it.