16

Yes, they are very different. Conceptually, a reflective measurement model happens when the indicators of a construct are considered to be caused by that construct. For example, an intelligence test: if you are more intelligent, you have a higher probability of getting the correct answer to a question. Hence your intelligence level is (theorized to) ...


13

Exploratory factor analysis (EFA) is appropriate (psychometrically and otherwise) for examining the extent to which one may explain correlations among multiple items by inferring the common influence of (an) unmeasured (i.e., latent) factor(s). If this is not your specific intent, consider alternative analyses, e.g.: General linear modeling (e.g., multiple ...


13

The accepted answer does not give models where the response is bounded between [0,1]. There are IRT models for exactly the case where the response variable is continuous, but bounded in this way. For example, Samejima [1] describes exactly this case. These models are sometimes called Continuous Response Models (CRM), and the case they're addressing is ...


13

The following two papers discuss cut-off values for reliability indices: Lance, C.E., Butts, M.M., & Michels, L.C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods, 9 (2), 202-220. Henson, R.K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on ...


11

The $\omega_h$ (hierarchical) coefficient gives the proportion of variance in scale scores accounted for by a general factor (1,2), usually from a second-order factor analysis. However, if any zero-order dimensions are reflected in such scales, $\omega_h$ will be less than Cronbach's $\alpha$ (which should only be used with unidimensional scales in any case)....


10

Two main causes: Small sample size. Even if the assumptions are met and the reliability is decent, an estimate computed from a particular sample can be negative, just as a sample mean is not equal to the population mean. Somewhat surprisingly, whereas experimental psychologists tend to be obsessed with statistical testing and are conditioned to ask a p-...


10

Although I feel a little sheepish contradicting both a "respected text" as well as another CV user, it seems to me that the Spearman-Brown formula is not affected by having items of differing difficulty. To be sure, the Spearman-Brown formula is usually derived under the assumption that we have parallel items, which implies (among other things) that the ...


9

I built on @Andy W's R-code and hope my changes are useful to someone else. I mainly changed it, so that it obeys the new syntax (no more opts) in ggplot2, so no more warnings adds the correlations as text now correlation text size reflects its effect size colour scheme shows the type of correlation (hetero/mono-trait/method). put the legend in the empty ...


9

That's pretty normal. CFA is a much more stringent criterion than EFA. EFA attempts to describe your data, but CFA tests if the model is correct. One reason for non-convergence is low average correlations (but then I'd expect RMSEA to be better). The chi-square test is essentially a test that your residuals are equal to zero, and RMSEA, TLI and CFI are ...


9

Both classical test theory (CTT) and item response theory (IRT) can provide guidance as far as which items are contributing to the latent trait you wish to measure, and which do not. With CTT, consider 1) item difficulty, 2) item correlation to total score, 3) item variance, and 4) impact on internal consistency estimates (e.g., Cronbach's alpha) if the ...


9

There are two equivalent ways to express the parallel analysis criterion. But first I need to take care of a misunderstanding prevalent in the literature. The Misunderstanding The so-called Kaiser rule (Kaiser didn't actually like the rule if you read his 1960 paper) eigenvalues greater than one are retained for principal component analysis. Using the so-...


8

According to mathworld: de Moivre developed the normal distribution as an approximation to the binomial distribution, and it was subsequently used by Laplace in 1783 to study measurement errors and by Gauss in 1809 in the analysis of astronomical data (Havil 2003, p. 157). Some extensions to the normal distribution have been developed in psychometry, ...


8

You can see what it means by studying the formula: $$ \alpha = \frac{K}{K-1}\left(1-\frac{\sum \sigma^2_{x_i}}{\sigma^2_T}\right) $$ where $T=x_1 + x_2 + ... x_K$. $T$ is the total score of a test with $K$ items, each scores $x_i$, respectively. Unpack the formula, using what we know about the covariance of a sum of RV's. If the test items are ...


8

As far as I can tell you are describing a partially crossed design. The good news is that this is one of Doug Bates's main development goals for lme4: efficiently fitting large, partially crossed linear mixed models. Disclaimer: I don't know that much about Rasch models nor how close a partially nested model like this gets to it: from a brief glance at ...


7

Cronbach’s alpha depends on the assumption that each indicator variable contributes equally to the factor, i.e., all (unstandardized) loadings must be the same (tau-equivalence). If this assumption is violated, true reliability will be underestimated. The second assumption for alpha is that the error variances of the indicators must be uncorrelated. In ...


7

A good approach to this kind of problem can be found in section 4 of the paper The Bayesian Image Retrieval System, PicHunter by Cox et al (2000). The data is a set of integer outcomes $A_1, ..., A_N$ where $N$ is the number of trials. In your case, there are 3 possible outcomes per trial. I will let $A_i$ be the index of the face that was left out. The ...


7

First, I would see if the doctors agree with each other. You can't analyze 50 doctors separately, because you'll overfit the model - one doctor will look great, by chance. You might try to combine confidence and diagnosis into a 10 point scale. If a doctors says that the patient doesn't have cancer, and they are very confident, that's a 0. If the doc says ...


6

You can calculate the reliability of your items from the CFA. From your standardized solution, calculate: (L1+...Lk)*2/[(L1+...Lk)*2+(Var(E1)+...+Var(Ek))] This will give the composite reliability, which should be close to alpha. It's harder to have good fit if you have high alpha, and it's harder to have high alpha if you have good fit. The extreme ...


6

Cronbach's $\alpha$ tends to increase with the number of questions. If after deleting an item your Cronbach's $\alpha$ is the same, then you haven't gained any reliability by having that item. All other things being equal we tend to prefer shorter questionnaires... so we drop the item. One possible issue here is that the "Cronbach's $\alpha$ if deleted" ...


5

Looks like I'm quite late to the game here, but the mirt package package can estimate WLE scores for dichotomous and polytomous models. You start by fitting, say, a graded response model to your data (or whatever your model may be, PCM, generalised PCM, nominal, rating scale, etc; see ?mirt for the possible options) then compute either a table summary of the ...


5

In following up on @caracal's reference suggestions, I found an almost-direct answer (no, these two rating systems are not equivalent if presented as number options to respondents) from Schwarz, Knäuper, Hippler, Noelle-Neumann, and Clark (1991). They present data on responses to the question, "How successful have you been in life, so far?" One version gave ...


5

As I stated in the comments above, missing data can be handled by either the ltm or mirt package when the data is MCAR. Here is an example of how to use both on a dataset with missing values: > library(ltm) > library(mirt > set.seed(1234) > dat <- expand.table(LSAT7) > dat[sample(1:(nrow(dat)*ncol(dat)), 150)] <- NA > head(dat) ...


5

From wikipedia, Kuder–Richardson Formula 20: "In statistics, the Kuder–Richardson Formula 20 (KR-20) first published in 1937 is a measure of internal consistency reliability for measures with dichotomous choices. It is analogous to Cronbach's α, except Cronbach's α is also used for non-dichotomous (continuous) measures."


5

You will rarely encounter the word “validated” in the psychometrics literature. Current thinking is that validation is an on-going process on which many type of evidence can be brought to bear. The validity of a particular measurement instrument also depends on the interpretations you want to draw from it and the specific population you are studying (see e.g....


5

The answer is (C), because that is exactly what the question stated: Higher scores indicate higher levels of depression. If this is a valid measure, then those who score higher are in fact more depressed than those who score low. Your reasoning for (A) is not correct. A valid measure is not necessarily reliable, but more importantly, a valid measure does ...


5

Probably some combination of both. Have the (predictive) tests been validated as measuring the same constructs as "the real test"? Are there three different tests (types) or is it the same test three times (tokens)? If it is the same test three times, the students may get better at it by virtue of exposure, rather than by having learned anything more. ...


5

It's not easy to say. First, the Spearman-Brown assumes that test items (or raters) are randomly sampled from a population of test items (or raters). This is never really true, particularly of tests, because making up more items is hard, and it's likely that you'll use the better items to start with - then you'll find that the test needs to be longer, so ...


5

As far as I understand, standard alpha() function from the psych package (http://personality-project.org/r/html/alpha.html) is not appropriate here, but I'm not absolutely sure about that. It seems to me that, in order to calculate Cronbach's $\alpha$ in R for mixed data, including polytomous items, you can use either function scoreItems() from the psych ...


5

Have you considered Item Response Theory-based methods? IRT is designed especially for this kind of purposes. Simple example is Rasch model that lets you compute both the student abilities and question difficulty in a single generalized linear model (or generalized linear mixed model). With binary answer format the Rasch model could be written as $$P(X_{ij}...


5

The issue you're alluding to is the 'approximate unidimensionality' topic when building psychological testing instruments, which has been discussed in the liturature quite a bit in the 80's. The inspiration existed in the past because practitioners wanted to use traditional item response theory (IRT) models for their items, and at the time these IRT models ...


Only top voted, non community-wiki answers of a minimum length are eligible