4
$\begingroup$

Questionnaires often contain questions which are similar but worded differently. Usually these questions are located in different parts of the questionnaire, intended to weed out participants who don't take the questionnaire seriously, but I've seen examples where related questions are placed together in a block, as well.

What is the purpose of these questions and, more importantly, how are they analysed?

For example, if the purpose of the survey is to find correlations between different variables on a Likert scale, then how can one go about analysing the data?

Assuming that simple linear regression is appropriate, for example, does one simply try out different 'versions' of the regressor (i.e. the different versions of the same question) and see which one results in the smallest p-value (for $H_0$: $\beta_1$ = 0, $H_A$: $\beta_1$ > 0)?

Or is there some other way to go about this?

Edit: To clarify, I am NOT asking how to conduct hypothesis tests on Likert scales or when I can assume them to be interval rather than ordinal, etc., nor am I asking for how to deal with multiple variables. My question was specifically asking about how similar questions on a questionnaire are dealt with, e.g. if Question 1 is Are you satisfied with our services?, Question 2 is Are you happy about our services?, and Question 3 is How good do you think our services are?

Edit 2: In fact, I don't mind if I get a answer about using logistic regression on true-or-false questions or something. My focus is on repeated questions, not on linear regression or the Likert scale.

Edit 3: Imagine I have four questions like these:

  1. Do you like our services? No 1 2 3 4 5 Yes
  2. Are you satisfied with our services? No 1 2 3 4 5 Yes
  3. Do you think our services are good enough? No 1 2 3 4 5 Yes
  4. Will you come to our store again? No 1 2 3 4 5 Yes

and I want to know the relationship between how people like our services and whether they'll be back. (Assume also that I'm sure simple linear regression is applicable. I've used each of 1-3 as regressor, found the least-square estimates of $b_0$ and $b_1$ each time, ensured the error is normal with the chi-squared goodness-of-fit test, ensured the relationship is linear with the F-test for lack of fit, dealt with outliers, etc.)

How should this relationship be analysis? If the responses to the first three questions are distributed very differently (say, significantly different mean or variance through Student's t- and F-tests), what is to be done about it? What if $H_0$ can be rejected for one or two questions but not the others - what can be done about this? Thanks.

$\endgroup$
  • 4
    $\begingroup$ Possible duplicate of Under what conditions should Likert scales be used as ordinal or interval data? $\endgroup$ – EdM Jan 1 '17 at 16:00
  • 1
    $\begingroup$ I think that a reason you may not have considered for near repeat questions is to check consistency of the answers. Sometimes participants are seriously trying to answer the questions but interpret similar questions differently and the answers may not seem to be consistent to the questionnaire designer. I think you have a potentially interesting question here and I notice that you are editing it to try to satisfy the comments of EdM. If your question seems to be only about questionnaire design it may be considered off topic for the site. $\endgroup$ – Michael R. Chernick Jan 1 '17 at 17:53
  • 1
    $\begingroup$ You may be able to satisfy EdM and the people who voted to close if you can provide a technical statistical question that doesn't duplicate other answers on the site. $\endgroup$ – Michael R. Chernick Jan 1 '17 at 17:55
  • 1
    $\begingroup$ I think you have explained that you don't want answers like the ones EdM referenced. The key is a question with the requirement I cited in my previous comment. $\endgroup$ – Michael R. Chernick Jan 1 '17 at 18:00
  • 2
    $\begingroup$ Thanks for taking the effort to show the statistical background for your question. I have retracted my vote to close. $\endgroup$ – EdM Jan 1 '17 at 21:05
4
$\begingroup$

When there are multiple questions getting at the same issue in a questionnaire, these are typically aded together to get a combined score. The individual questions are Likert items, with their combination providing a Likert scale. This helps get around several types of problems that can arise from depending too heavily on a single response to a single item. So the task of a designer of a questionnaire is more typically to evaluate how well a set of items all address the same issue, rather than to find a single "best" item as this question might suggest. I suppose you could consider this a case of making multicollinearity work for you. Cronbach's alpha is a standard way to evaluate whether a set of items is measuring the same thing, to evaluate their internal consistency.

As Michael Chernick has noted in another answer, any plan to choose a "best" item based on p-values for relations to your outcome variable is fraught with difficulty. In addition to the multiple-testing statistical issues, it is likely that the "best" item found in one sample of answers to a set of items will be different from that found in another sample. Try repeating your item-selection protocol on multiple bootstrap samples of your data to illustrate that problem.

This page is one of several useful threads on this site about design, validation, and interpretation of Likert items and scales; a search for Likert item shows many more. I understand the motivation to try to use a smaller number of questions, but that can pose problems as discussed on this page.

If you have several items that you want to analyze as individual predictors rather than in a combined Likert scale, logistic regression as suggested in the question would be a possible approach, but with some cautions. The responses to these items are likely to have substantial multicollinearity, which can pose problems for any type of regression. Ridge regression is often a useful way to combine multicollinear predictors without running risks of overfitting, in logistic, linear, or Cox regression models. As with other regressions, you could incorporate transformations of the predictor variables if needed to get around issues arising from the distributions of the predictors or, more precisely, of their relations to outcome.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I think between the two of us we have made a good effort to salvage this question and provide informative answers. I am sure the OP is appreciative. $\endgroup$ – Michael R. Chernick Jan 1 '17 at 23:36
  • $\begingroup$ Thanks to both of you! I indeed am, and will look up the work you mentioned in greater detail, as I'm not familiar with all of them. $\endgroup$ – WavesWashSands Jan 2 '17 at 15:08
  • 1
    $\begingroup$ +1, note however that Cronbach's alpha cannot be used "to evaluate whether a set of items is measuring the same thing" (see here). $\endgroup$ – gung - Reinstate Monica Jan 3 '17 at 0:49
2
$\begingroup$

I have spoken about the design issue (use of similar near repeat questions in the questionnaire) in comments. I think I can give an answer to the analysis question (not necessarily complete). I will address the multiple testing issue.

In general when doing hypothesis testing whether in regression or not the use of multiple hypothesis tests inflates the p-value. There is a lot of work on this issue and if you search for p-value adjustment here or on google, you will find numerous posts and a great deal of literature on the subject.

I have contributed to a number of posts on this topic. I like to refer people to the text by Westfall and Young published in 1993 by Wiley on resampling methods for p-value adjustment. The subject of simultaneous inference (another name for it) or multiple testing goes back a long way. I know that work by John Tukey and others goes back at least to the 1940s.

I think my answer here is in the spirit of addressing your analysis question. I am speculating that what you are suggesting is that if two questions gives conflicting answers, you try a hypothesis test with one of the conflicting questions and its answer left out and the other question and answer kept in. Then you make a second try at the same hypothesis test switching the question and answer left in with the question and answer left out. This would be an example where multiple testing methodology could be used.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.