General validity/reliability of questionnaires I have a general question concerning the reliability/validity of questionnaires. I'm working in Human Resources and we plan to conduct a workplace engagement survey. I attended a conference and a company presented their questionnaire on workplace engagement. They explained that they did research within many different organizations, and based on their analysis (Confirmatory Factor Analysis) they came up with 7 drivers of engagement. Each driver/theme is made  of various questions.
My question is, if we would choose to use their questionnaire for our workplace engagement survey, could I then assume that the identified themes would be valid for our organization as well? The research has been done within different organization, however, our organization is a university. So I wonder if using this questionnaire might not give me valid and reliable results.
Many thanks for all your help in advance,
Tanya
 A: The short answer is maybe, but the probability is pretty low without more information.  I will try to keep this as non-technical as possible.
To answer this question in detail for yourself, there are a few questions that you need to either research (a Google Scholar search for the instrument or read the company white paper) or ask the company representative offering the instrument directly (which is least preferable):


*

*How many times has this instrument been used in research studies?  The number should be for studies not individuals who used the instrument.  The reason this is important is if it was used in only one or a few research studies, there isn't much evidence of either reliability or validity.  Think of a web survey that is posted on Facebook - it may have been answered hundreds of thousands of times, but there is no evidence of validity or reliability.

*What is the intended population for the study and/or what research sample populations has the instrument been used for? If this instrument was developed for large manufacturing companies in China and you are using it for a small college in the United States, there may not be much external validity (different populations).

*What were the "7 drivers of engagement" based on or developed from?  An example from my leadership studies: there are thousands of "10 successful leadership traits" type articles, websites and books - but they are not research based or empirically tested.  They are lists of "yeah that sounds good"s or anecdotal experiences that worked for a certain leader in a certain company at a certain point of the business cycle.
There are also a couple of questions you should ask yourself and the stakeholders before starting this project:


*

*Why are we concerned about workplace engagement?  And probably a better question would be: what part of workplace engagement are we concerned about? There are over 600,000 studies on workplace engagement (Google Scholar), so what part of it are the you and stakeholders interested in and why?

*What will we do with the information we collect?  This actually goes back to workplace surveys in general - what are you going to do with this information (use it to set up new programs/policies) or is someone going to collect it and not use it for action or improvement?

*Look at the questions of the survey and ask "is this the type of information we need to know about our employees' engagement?  This goes along the lines of face validity, but from a user point of view.  If the questions aren't relevant for your organization, why use them?
One last item that incorporates all 3 questions above, if you are looking for the answer to a specific question or a problem, you could make your own survey (Survey Monkey or some other site) and ask the questions you need answers too.  Yes it isn't "scientifically" validated or tested for reliability, but it may be the most effect solution for your situation.
I am sorry of the length, but I have been in your shoes and wished someone would have given me some free advice.  
A couple of good free resources for reliability and validity:
Research Methods Knowledge Base (http://www.socialresearchmethods.net/kb/relandval.php) 
John Hopkins website (http://ocw.jhsph.edu/courses/hsre/pdfs/hsre_lect7_weiner.pdf)  
