2
$\begingroup$

I am doing research on school security. Questionnaire pertains to questions regarding availability or non availability of certain items so as to reach on conclusion that what is available and what is to be provided to complete the security package.

There are 57 questions and almost all are "yes" or "no" with few exceptions of answers as "not applicable" or some "range of amount in currency" like "less than $5000" and "Less than $ 10000". Now, how do I check the reliability. Someone said that since it is quantitative so "Cronbach's Alpha". But once I apply that it discard many questions and index is never more than 0.5. Please help and guide.

In addition, I have consulted few statisticians and they said that the questionnaire I have is just like a checklist which is conformation of the instructions given to schools by the department. So the answer is just yes and no. In this case reliability is not required to be checked.

Improve this question
$\endgroup$
9
  • $\begingroup$ At the same time I have another questionnaire in which I am asking the security experts that which item is immediately required, which is later required and which is not required. So they tick appropriate column in front of item name. What do I have to do with this to check reliability? $\endgroup$
    – Afaq
    Commented Aug 31, 2016 at 20:16
  • $\begingroup$ Afaq, it would help if you told us more about what type of person would answer your questionnaire. Are you hoping to give it to security experts in order to assemble a consensus on what schools should do to become more secure? Or do you already know what a complete 'security package' consists of, and plan to give the questionnaire to school administrators to find out what their security gaps are? Understanding your intent better would enable CV users to answer your question most helpfully. $\endgroup$
    – David C. Norris
    Commented Aug 31, 2016 at 20:32
  • $\begingroup$ First of all as per the state given policy, we already know what is to be provided. So I will go about in a following way: (1) I will ask school heads that what all they already have out of the list given by the state and also what funds they have got or not got. (2) Since Schools have very less equipment and less funds as well so I will ask few security specialists the priority of procurement to give a package for Minimum, Acceptable and Ideal....... Now I have a sample of almost 300 School Heads and 6 Security Specialists. I hope I have made it clear. If not you may ask more. $\endgroup$
    – Afaq
    Commented Sep 1, 2016 at 8:14
  • $\begingroup$ School Head questionnaires have two options i.e. Yes and No. In the questions where state of funds was asked they were asked about amount they have. $\endgroup$
    – Afaq
    Commented Sep 1, 2016 at 8:55
  • $\begingroup$ That does clarify your intent to some extent, thanks. But now I wonder what you mean by 'reliability'. Do you mean correctness of the answers? For example, are you looking for ways to detect that a school actually has a lockable gate, even when the questionnaire response is (incorrectly) given as 'NO'? $\endgroup$
    – David C. Norris
    Commented Sep 1, 2016 at 15:23

3 Answers 3

Reset to default
1
$\begingroup$

I would encourage you to maintain your focus on your scientific questions, and select your statistical techniques appropriately to those questions. Issues like the 'reliability' of questionnaire items should be explored with particular reference to your realistic concerns about the information they yield.

It sounds to me as if you believe your questionnaire items ask for straightforward factual information from competent people who are in a position to know the correct answers. But maybe your supervisor has thought of something you've missed; it would pay to ask. Unless you (or your supervisor) can tell a reasonable story about how one or more of the items might give unreliable information, then you have no basis for choosing any statistical technique for assessing 'reliability'.

The higher your ratio of thinking to statistics, the more useful your work is likely to be. There are few things that more clearly flag poor social science research than the desultory, mechanical application of 'obligatory' statistical tests (or other procedures) in the absence of genuine scientific motivation. As one example of where a low $thinking:statistics$ ratio can lead, consider this cautionary tale.

If your site visits showed that there were any items that were not answered as reliably as you hoped, and if you have questions about how to present or analyze your data about those specific reliability problems, then you may wish to ask a new CV question.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I think you are getting my point. My questionnaire is based on a document and answers can only be yes or no. So I think if the answer is to be in this way like a fact that, "Is the wall 8 feet High", "Do you have an Armed Guard", "Do you get special grant for Security", "Do you have CCTV System". I suppose in such questions reliability check is not needed. Its descriptive not infrential. What do you say? $\endgroup$
    – Afaq
    Commented Sep 5, 2016 at 12:52
  • $\begingroup$ I'm willing to go out on a limb at this point, and guess that your supervisor was hoping that some statistical 'Validation Fairy' might flutter in to sprinkle magic 'validation dust' on your survey. The short answer is, there is no Validation Fairy. No Santa Claus, no Easter Bunny, no Validation Fairy. Please don't forget to 'accept' an answer if it has worked for you. $\endgroup$
    – David C. Norris
    Commented Sep 6, 2016 at 20:25
  • $\begingroup$ None of the answer has solved my query, so how can I select an answer. $\endgroup$
    – Afaq
    Commented Sep 8, 2016 at 18:22
  • 1
    $\begingroup$ You might consider refining your question in that case, Afaq, so that it elicits answers that are more useful to you and directed toward your needs. One thing that might help is searching for related Q&A's here on CV. Seeing how some successfully answered questions were asked might help you rephrase. Also, consider doing some research on your own, and including links to what you've read in the text of your question. $\endgroup$
    – David C. Norris
    Commented Sep 9, 2016 at 18:08
0
$\begingroup$

There is no need to check the reliability of demographic type data like (range of amount etc..). While You can check the reliability of rest of items. It looks like you have a dichotomous (Yes/No, True False) data. In this case, You can either use Cronbach Alpha or KR-20 test both will produce identical results.

And If Index is not more than .50, it does not mean that there is issue in process. There may be issues in items.

Firstly, check the demographics of all items (minimum and maximum values), check weather the range of each item's values are correct.

Secondly, check weather there is any item with negative wordings. Thirdly, generate the correlation matrix of all items, You must have some very low or negative correlated items.

Lastly, If I were you I would closely observe each items and group them into categories (factor) then check the reliability of factor (subscale).

Improve this answer
$\endgroup$
0
0
$\begingroup$

You might want to think about the possibility that your questionnaire is assessing different latent constructs. I suggest you factor analyze your data to determine what those constructs are. This would fall more in the realm of validity but you may want to consider this to see how well your items work together (broadly speaking, of course).

With regard to reliability- have you considered all the factors that influence a measure's reliability? For example... 1) test length, 2) time given to participants to complete the measure, 3) item difficulty or clarity, 4) intervals between tests (if you are looking at test-retest reliability, and 5) the environment in which your participants are answering your questionnaire. Something else to consider would be to examine how your participants' characteristics are influencing their responses. Participant homogeneity is another major factor that influences a measure's reliability.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.