How are questions of similar meaning in questionnaires dealt with?

Question

Questionnaires often contain questions which are similar but worded differently. Usually, these questions are located in different parts of the questionnaire, intended to weed out participants who don't take the questionnaire seriously, but I've seen examples where related questions are placed together in a block, as well.

What is the purpose of these questions and, more importantly, how are they analysed?

For example, if the purpose of the survey is to find correlations between different variables on a Likert scale, then how can one go about analysing the data?

Assuming that simple linear regression is appropriate, for example, does one simply try out different 'versions' of the regressor (i.e. the different versions of the same question) and see which one results in the smallest p-value (for $H_0$: $\beta_1$ = 0, $H_A$: $\beta_1$ > 0)?

Or is there some other way to go about this?

Edit: To clarify, I am NOT asking how to conduct hypothesis tests on Likert scales or when I can assume them to be interval rather than ordinal, etc., nor am I asking how to deal with multiple variables. My question was specifically asking about how similar questions on a questionnaire are dealt with, e.g. if Question 1 is: Are you satisfied with our services?, Question 2 is: Are you happy about our services?, and Question 3 is: How good do you think our services are?

Edit 2: In fact, I don't mind if I get an answer about using logistic regression on true-or-false questions or something. My focus is on repeated questions, not on linear regression or the Likert scale.

Edit 3: Imagine I have four questions like these:

1. Do you like our services? No 1 2 3 4 5 Yes
2. Are you satisfied with our services? No 1 2 3 4 5 Yes
3. Do you think our services are good enough? No 1 2 3 4 5 Yes
4. Will you come to our store again? No 1 2 3 4 5 Yes

and I want to know the relationship between how people like our services and whether they'll be back. (Assume also that I'm sure simple linear regression is applicable. I've used each of 1-3 as a regressor, found the least-square estimates of $b_0$ and $b_1$ each time, ensured the error is normal with the chi-squared goodness-of-fit test, ensured the relationship is linear with the F-test for lack of fit, dealt with outliers, etc.)

How should this relationship be analysed? If the responses to the first three questions are distributed very differently (say, significantly different mean or variance through Student's t- and F-tests), what is to be done about it? What if $H_0$ can be rejected for one or two questions but not the others - what can be done about this? Thanks.

Answer 1

When there are multiple questions getting at the same issue in a questionnaire, these are typically aded together to get a combined score. The individual questions are Likert items, with their combination providing a Likert scale. This helps get around several types of problems that can arise from depending too heavily on a single response to a single item. So the task of a designer of a questionnaire is more typically to evaluate how well a set of items all address the same issue, rather than to find a single "best" item as this question might suggest. I suppose you could consider this a case of making multicollinearity work for you. Cronbach's alpha is a standard way to evaluate whether a set of items is measuring the same thing, to evaluate their internal consistency.

As Michael Chernick has noted in another answer, any plan to choose a "best" item based on p-values for relations to your outcome variable is fraught with difficulty. In addition to the multiple-testing statistical issues, it is likely that the "best" item found in one sample of answers to a set of items will be different from that found in another sample. Try repeating your item-selection protocol on multiple bootstrap samples of your data to illustrate that problem.

This page is one of several useful threads on this site about design, validation, and interpretation of Likert items and scales; a search for Likert item shows many more. I understand the motivation to try to use a smaller number of questions, but that can pose problems as discussed on this page.

If you have several items that you want to analyze as individual predictors rather than in a combined Likert scale, logistic regression as suggested in the question would be a possible approach, but with some cautions. The responses to these items are likely to have substantial multicollinearity, which can pose problems for any type of regression. Ridge regression is often a useful way to combine multicollinear predictors without running risks of overfitting, in logistic, linear, or Cox regression models. As with other regressions, you could incorporate transformations of the predictor variables if needed to get around issues arising from the distributions of the predictors or, more precisely, of their relations to outcome.

Answer 2

I have spoken about the design issue (use of similar near-repeat questions in the questionnaire) in the comments. I think I can give an answer to the analysis question (not necessarily complete). I will address the multiple testing issue.

In general when doing hypothesis testing whether in regression or not the use of multiple hypothesis tests inflates the p-value. There is a lot of work on this issue and if you search for p-value adjustment here or on google, you will find numerous posts and a great deal of literature on the subject.

I have contributed to a number of posts on this topic. I like to refer people to the text by Westfall and Young published in 1993 by Wiley on resampling methods for p-value adjustment. The subject of simultaneous inference (another name for it) or multiple testing goes back a long way. I know that work by John Tukey and others goes back at least to the 1940s.

I think my answer here is in the spirit of addressing your analysis question. I am speculating that what you are suggesting is that if two questions give conflicting answers, you try a hypothesis test with one of the conflicting questions and its answer left out and the other question and answer kept in. Then you make a second try at the same hypothesis test switching the question and answer left in with the question and answer left out. This would be an example where multiple testing methodologies could be used.