Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

As part of an advisory team in my educational institution for undergraduate nurses, I (among others) have been assigned to design a questionnaire that will assess students' and academic staff's perceptions about the ability of the final year's exams as an adequate tool for evaluating students': knowledge-understanding /competence-skills/ and judgment-approach (in other words, to judge whether they are ready to safely work in the real world). The final questionnaire comprises 21 questions regarding the above mentioned 3 basic concepts. I proposed the answers to be given on a 5-point balanced Likert-type scale (strongly disagree to strongly agree with a neutral midpoint). However, the rest of the team is in favour of using an unbalanced 4-point scale (disagree/agree a little/agree/strongly agree).

Question 1: Which scale would be more appropriate?

Also, I believe that this proposed tool needs to be validated and checked for reliability first before being administered to the target population, however, most of my colleagues disagree, since, the questions arise from the corresponding 21 statements of our national regulatory board, according to which:

  1. "Nurses must demonstrate knowledge of planning of health care measures",
  2. "Nurses must demonstrate the ability to draw care plans",
  3. "Nurses must demonstrate the ability to apply their knowledge so as to deal with different situations" and so on.

The majority of the team is in favour of converting these statutory statements into a questionnaire as follows:

I believe that the test: Q1 "demonstrated knowledge of planning of health care measures", Q2 "demonstrated the ability to draw care plans"

and so on up to question 21, without checking for validity and reliability first.

Question 2: Shouldn't this tool be tested first for its reliability and validity (convergent, discriminant, criteria validity, factor analysis, etc)?

share|improve this question

1 Answer 1

Q1. 4 or 5 point scale (strongly disagree to strongly agree with or without a neutral midpoint)

A1. I think the use of even or odd number of scale points is not a matter that has a definitive answer. There are arguments on both sides of this question. Since you want a yes-no answer, the 4 point scale may be better suited to your purpose than a scale with a neutral mid-point.

Parenthetically, I would suggest that a convincing evaluation additionally would include a detailed evaluation of the actual content of each of the final exam questions such as: "Do you think a minimally competent nurse would answer this question incorrectly?" (This is the type of approach developed by Angoff at ETS many years ago. See, this secondary source, for example.) Global opinions are open to halo and other types of bias.

Q2a. Should the reliability and convergent and discriminant validity be evaluated before the scale is used?

A2a. Inter-rater agreement might be evaluated after the data is collected. If inter-rater agreement is low, the reasons can be probed. If it turns out the survey has to be repeated due to low agreement, it is not a high cost undertaking. Convergent and discriminant validity are evaluated based on correlations. However, in your case, these correlations may be driven more by how hard individual nursing students studied, how bright they are, etc. than by the design of the survey.

Q2b. Should the criterion validity of the scale be evaluated before the scale is used?

A2b. Criterion validity requires a criterion and you do not have one (or you would not be undertaking this survey type evaluation). The best that is usually done in this type of undertaking is content validity. (See for example, the AERA/APA Standards, soon to be revised.)

Q3. Designing the questionnaire around the 21 statements of your national regulatory board.

A3. This is a great idea since it builds on an accepted statement of standards. Do you have alternatives to suggest?

share|improve this answer
    
+1, this is a really nice answer. There is a lot of solid information, clearly stated & nicely organized. (You may want to add another <enter> before A3, though.) –  gung Aug 20 '12 at 21:18
    
Thank you for your answer. My concern has to do with the fact that the 4-point scale is unbalanced (one negative and three positive answers) something that could inevitably lead to more positive scoring. –  dino Aug 21 '12 at 9:00
    
Also, don't you think that factor analysis should be performed? –  dino Aug 21 '12 at 9:08
    
(1) The anchors on the 4 point scale you report are clear. This addresses your concern about unbalanced responding. Perhaps it would be better to have two negative and two positive scale points, but I do not think the 4 point scale you described is a flaw. (2) A factor analysis is based on correlations, so see my comment above on correlations. What might a factor analysis indicate that would concern you? –  Joel W. Aug 21 '12 at 18:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.