Validating questionnaires I am designing a questionnaire for my dissertation.  I am in the process of validating the questionnaire I have applied a Cronbach's alpha test to the initial sample group.  The responses to the questionnaire are on a Likert scale; can anyone suggest any further tests to apply to help test its validity. I am not an expert on statistics so any help would be appreciated.
I have been doing some research and it appears I can do a Rasch analysis has anyone got any free software sites to apply this test and advice?
 A: I will assume that your questionnaire is to be considered as one unidimensional scale (otherwise, Cronbach's alpha doesn't make very much sense). It is worth running an exploratory factor analysis to check for that. It will also allow you to see how items relate to the scale (i.e., through their loadings).
Basic steps for validating your items and your scale should include:


*

*a complete report on the items' basic statistics (range, quartiles, central tendency, ceiling and floor effects if any);

*checking the internal consistency as you've done with your alpha (best, give 95% confidence intervals, because it is sample-dependent);

*describe you summary measure (e.g., total or mean score, aka scale score) with usual statistics (histogram + density, quantiles etc.);

*check your summary responses against specific covariates which are supposed to be related to the construct your are assessing -- this is referred to as known-group validity;

*if possible, check your summary responses against known instruments that purport to measure the same construct (concurrent or convergent validity).


If your scale is not unidimensional, these steps have to be done for each subscale, and you could also factor out the correlation matrix of your factors to assess the second-order factor structure (or use structural equation modeling, or confirmatory factor analysis, or whatever you want). You can also assess convergent and discriminant validity by using Multi-trait scaling or Multi-trait multi-method modeling (based on interitem correlations within and between scales), or, again, SEMs.
Then, I would say that Item Response Theory would not help that much unless you are interested in shortening your questionnaire, filtering out some items that show differential item functioning, or use your test in some kind of a computer adaptive test. 
In any case, the Rasch model is for binary items. For polytomous ordered items, the most commonly used models are :


*

*the graded response model

*the partial credit model

*the rating scale model.


Only the latter two are from the Rasch family, and they basically use an adjacent odds formulation, with the idea that subject has to "pass" several thresholds to endorse a given response category. The difference between these two models is that the PCM does not impose that thresholds are equally spaced on the theta (ability, or subject location on the latent trait) scale. The graded response model relies on a cumulative odds formulation. Be aware that these models all suppose that the scale is unidimensional; i.e., there's only one latent trait. There are additional assumptions like, e.g., local independence (i.e., the correlations between responses are explained by variation on the ability scale). 
Anyway, you will find a very complete documentation and useful clues to apply psychometric methods in R in volume 20 of the Journal of Statistical Software: Special Volume: Psychometrics in R. Basically, the most interesting R packages that I use in my daily work are: ltm, eRm, psych, psy. Others are referenced on the CRAN task view Psychometrics. Other resources of interest are: 


*

*Notes on the use of R for psychology experiments and questionnaires

*Using R for psychological research (W. Revelle is actually writing a book on psychometrics in R)

*the PsychoR project (it does not focus on IRT and scale development, though).


A good review on the use of FA vs. IRT in scale development can be found in Scale construction and evaluation in practice: A review of factor analysis versus item response theory applications, by ten Holt et al (Psychological Test and Assessment Modeling (2010) 52(3): 272-297).
A: While supporting everything said above, i would suggest that you do the following (in similiar enough order)
Firstly, you should be using R, if not you should start. The following advice is predicated on the use of R.
I'll assume that you have, at this point, calculated the descriptive statistics et al. If not, the psych package has a describe() function which should give you the stats you need. 
Install the psych package from CRAN. 
Load the psych package. 
Use the fa.parallel routine on your data. This should give you a number of factors to retain. 
Then, use the VSS(routine). This calculates the MAP criterion which gives you a different (normally) number of factors to retain. 
Use a form of factor analysis (not principal components) and an oblique rotation for each number of factors. If your factors do not appear to be correlated after an oblique rotation, switch to orhogonal rotation. This is as an orthogonal structure can be determined from an oblique rotation, but not vice versa. 
Extract all the factor solutions between the MAP criterion and the parallel analysis criterion. Determine which of these has the best fit indices and makes the most sense. This is the one you should retain. 
On IRT, having used both ltm and eRm, I would suggest starting with eRm. It has better graphics functions for your models, and support for polytomous models is greater. That being said, it only fits Rasch models, and often data from psychological questionnaires do not meet the requirements for them. 
Good luck! Psychometrics is a lot of fun, as you will no doubt discover.  
A: Validating a questionnaire means proving it measures what it's supposed to measure. So I'd say, this is mostly not a statistical question and it cannot be answered without knowing your questionnaire's specific content. Cronbach alpha is not about validity, but about internal consistency, which is somewhat related to reliability (or one can say it is reliability presuming your questions are interchangeable -- but they're not).
So what could you do to validate your questionnaire? You could study which psychological processes give rise to specific pattern of results (e.g. by trying to induce such patterns with experimental manipulations, or using a think-aloud procedure ["protocol analysis", Ericsson & Simon, 1992]). Or compare some contrasting groups (e.g. patients with controls) that should have differing scores. Or correlate it with an external criterion that should be correlated with the trait you're measuring. Or measure the trait by the Psychoscope (TM) and use that as a criterion.
The other answers are more useful in pointing out what you probably can realistically do -- even though most of it is, strictly speaking, not about validity (exc. Chi's references to "known group validity" and external validity).
See also Markus & Borsboom (2013) for a modern approach to validity (this and some other useful refs @Borsboom's homepage). 
