Psychometrics has evolved as a subfield of psychology to become the science of measurement of unobservable individual characteristics.

learn more… | top users | synonyms

21
votes
2answers
633 views

Detecting patterns of cheating on a multi-question exam

QUESTION: I have binary data on exam questions (correct/incorrect). Some individuals might have had prior access to a subset of questions and their correct answers. I don’t know who, how many, or ...
15
votes
5answers
798 views

Introduction to structural equation modeling

I am asked by colleagues some help in this subject, that I don’t really know. They made hypotheses on the role of some latent variables in one study, and a referee asked them to formalize this in SEM. ...
14
votes
2answers
9k views

Inter-rater reliability for ordinal or interval data

Which inter-rater reliability methods are most appropriate for ordinal or interval data? I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and ...
14
votes
4answers
4k views

How to get started with applying item response theory and what software to use?

Context I have been reading about item response theory, and I find it fascinating. I believe I understand the basics, but I am left wondering how to apply statistical techniques related to the area. ...
11
votes
3answers
3k views

Validating questionnaires

I am designing a questionnaire for my dissertation. I am in the process of validating the questionnaire I have applied a Cronbach's alpha test to the initial sample group. The responses to the ...
11
votes
3answers
3k views

Factor analysis of questionnaires composed of Likert items

I used to analyse items from a psychometric point of view. But now I am trying to analyse other types of questions on motivation and other topics. These questions are all on Likert scales. My initial ...
11
votes
2answers
2k views

Assessing reliability of a questionnaire: dimensionality, problematic items, and whether to use alpha, lambda6 or some other index?

I am analyzing scores given by participants attending an experiment. I want to estimate the reliability of my questionnaire which is composed of 6 items aimed at estimating the attitude of the ...
10
votes
5answers
1k views

Is it appropriate to treat n-point Likert scale data as n trials from a binomial process?

I've never liked how people typically analyze data from Likert scales as if error were continuous & Gaussian when there are reasonable expectations that these assumptions are violated at least at ...
10
votes
2answers
187 views

Identification of useless questions from a questionnaire

I'm developing a questionnaire. To improve its reliability and validity I want to use statistical methods. I want to eliminate questions whose answers are always the same. This means that nearly all ...
9
votes
2answers
6k views

How to transform ordinal data from questionnaire into proper interval data?

Are there any straightforward methods of transforming ordinal level data into interval level (just as there are for doing it the other way round)? And performable in Excel or SPSS? Having the data, ...
9
votes
2answers
2k views

Recommended procedure for factor analysis on dichotomous data with R

I have to run a factor analysis on a dataset made up of dichotomous variables (0=yes, 1= no) and I don´t know if I'm on the right track. Using tetrachoric() I ...
9
votes
2answers
774 views

Where do the descriptors for Cronbach's alpha values come from (e.g., poor, excellent)?

It seems fairly common to describe Cronbach's alpha values as follows: α ≥ 0.9 Excellent 0.7 ≤ α < 0.9 Good 0.6 ≤ α < 0.7 Acceptable 0.5 ≤ α < 0.6 Poor α < 0.5 Unacceptable Where ...
8
votes
4answers
1k views

Can one validly reduce the numbers of items in a published Likert-scale?

[edits made in response to feedback- thanks :-) ] Doh! More edits! Sorry! Hello- I am doing some rather rough and ready data collection with a survey sent out to healthcare staff using a published ...
8
votes
4answers
394 views

Why aren't all tests scored via item analysis/response theory?

Is there a statistical reason why item analysis/response theory isn't more widely applied? For instance, if a teacher gives a 25 question multiple choice test and finds that 10 questions were answered ...
8
votes
2answers
2k views

How to reduce number of items using factor analysis, internal consistency, and item response theory in conjunction?

I am in the process of empirically developing a questionnaire and I will be using arbitrary numbers in this example to illustrate. For context, I am developing a psychological questionnaire aimed at ...
7
votes
4answers
8k views

How to compute correlation between/within groups of variables?

I have a matrix of 1000 observations and 50 variables each measured on a 5-point scale. These variables are organized into groups, but there aren't an equal number of variables in each group. I'd ...
7
votes
1answer
2k views

Looking for a step through an example of a factor analysis on dichotomous data (binary variables) using R

I have some dichotomous data, only binary variables, and my boss asked me to perform a factor analysis using the tetrachoric correlations matrix. I’ve previously been able to teach myself how to run ...
7
votes
2answers
666 views

Reliability in Elicitation Exercise

I am trying to calculate the reliability in an elicitation exercise by analysing some test-retest questions given to the experts. The experts elicited a series of probability distributions which were ...
7
votes
1answer
836 views

How to perform inter-rater reliability with multiple raters, different raters per participant, and possible changes over time?

Participants were rated twice, with the 2 ratings separated by 3 years. For most participants the ratings were done by different raters, but for some (< 10%) the same rater performed both ratings. ...
7
votes
1answer
272 views

What am I measuring when I apply a graded response model to the “Hunting of the Snark” dataset?

In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few ...
6
votes
3answers
2k views

Differences between tetrachoric and Pearson correlation

Currently I'm analysing around 300 items in the field of education. I'm interested in the dimensionality of the dataset. So I compute a matrix of tetrachoric correlation. The goal is to do a factor ...
6
votes
3answers
2k views

Is 0 a valid value in a Likert scale?

I have carried out my pilot study on language learning motivation using a 6 point Likert scale but from 0 (strongly disagree) to 5 (very much agree). I noticed a colleague in his survey used 1 to 6. ...
6
votes
2answers
955 views

Item Analysis for an R newbie

I am trying to assess a 20-item multliple choice test. I want to perform an item analysis such as can be found in this example. So for each question I want the P-value and the correlation with the ...
6
votes
3answers
904 views

Validating an existing questionnaire into another language

I am doing research and trying to use a questionnaire that is available in English in another language. Is there a systematic approach for validating a questionnaire in another language? What ...
6
votes
2answers
264 views

Estimating the probability of a person getting a question right

We need to estimate the probability of a person getting a question right for a given content area given his history of getting questions in the same content area right in the past. We would also ...
6
votes
1answer
202 views

EFA clearly supports one-factor, measure is internally consistent, but CFA has poor fit?

I am exploring the psychometric properties of a 10-item self-report measure. I have about 400 cases in two independent samples. The items are completed on 4-point Likert scales. An EFA clearly ...
6
votes
2answers
711 views

Advice on scientifically sound scale construction

I've been given a set of 20 Likert-items (ranging from 1-5, sample size n = 299) within the field of organizational research. The items are intended to measure a latent concept which is ...
6
votes
2answers
223 views

Reparameterizing the binomial link for psychometric data

Consider an experiment where a square and diamond appear on a screen, one before the other, and participants are required to judge which came first. Manipulating the time interval between the two ...
5
votes
4answers
6k views

What are the primary differences between z-scores and t-scores, and are they both considered standard scores?

We are currently converting student test scores in this manner : ( ScaledScore - ScaledScore Mean ) / StdDeviation ) * 15 + 100 I was referring to this ...
5
votes
4answers
675 views

FA: Choosing Rotation matrix, based on “Simple Structure Criteria”

One of the most important issues in using factor analysis is its interpretation. Factor analysis often uses factor rotation to enhance its interpretation. After a satisfactory rotation, the rotated ...
5
votes
1answer
5k views

Intraclass correlation coefficient vs. F-test (one-way ANOVA)?

I'm a little confused regarding the intraclass correlation coefficient and one-way ANOVA. As I understand it, both tell you how similar observations within a group are, relative to observations in ...
5
votes
3answers
4k views

Is it acceptable to use Cronbach's alpha to assess reliability of questionnaire composed of categorical and conditional items?

Background: I have data on a questionnaire made up of categorical factual items. Some of them are binary and others have more than two categories. For example: ...
5
votes
1answer
611 views

IRT in R: Does anyone know of an IRT item calibration function that can cope with NA's?

I have a dataset that looks like this. ...
5
votes
2answers
114 views

Sampling considerations in psychometric applications of item response theory

Background I have developed a recent interest in Item Response Theory and its applications. I am studying clinical psychology and am most interested in polytomous models aimed at modelling ...
5
votes
1answer
28 views

Item reliability or fit in IRT-based adaptive tests?

My background is in machine learning and statistics, but I am relatively new to psychometrics and testing. Nearly all of the literature I've found on item reliability refers to Chronbach's alpha or ...
4
votes
4answers
148 views

If a measure is valid (but not necesarily reliable), can it be consistently replicated?

I had this question on an exam, and I was positive that the answer was A. The Beck Depression Inventory is a scale intended to measure depression levels, with higher scores indicative of higher ...
4
votes
1answer
898 views

Which package to use for convergent and discriminant validity in R?

I am new to R and want to assess validity of some scales using R. Is there a package that I can use to test for convergent and discriminant validity in R?
4
votes
3answers
141 views

How to check psychometric type of scales?

It is well known, that there are diffrent types of scales (see Wikipedia Level of measurement). In psycholinguistics there is Semantic Differential technique. It uses a scale (usually 5 or 7 grades) ...
4
votes
1answer
114 views

Testing the importance of an item among a finite set of items

Consider the following survey question: Q: How would you classify the importance for you of the following 5 items: A B C D E Assign to each item a number ...
4
votes
1answer
210 views

Good internal factor structure but poor cronbach alpha?

I am running a CFA and getting good fit indices (CFI = .99, RMSEA = .01) for a uni-dimensional scale. However, when I test for internal consistency, I get poor Cronbach alphas (alpha = .6). I have ...
4
votes
1answer
123 views

IRT/Rasch modeling with very large N

I want to fit a 1-parameter IRT model on a questionaire with 15 questions and about six million people. Considering the large N, standard errors aren't essential. It looks like the IRT world is sort ...
4
votes
3answers
216 views

Are discrimination parameters in two-parameter IRT models only specific to items?

Discrimination parameters in two-parameter model from IRT are usually considered item parameters. But I've come to doubt it. Think about psychophysics; for example, detecting luminance. I don't ...
4
votes
3answers
369 views

$\chi^2$ tests to compare the fit of large samples logistic models

Does anyone know of any $\chi^2$ tests to compare the fit of logistic models which factor out the sample size? I'm dealing with a very large sample and I fear the significant $\chi^2$ test I get when ...
4
votes
1answer
2k views

4 point and 5 point Likert scales in a new questionnaire

I'm looking to confirm whether having a 4 point Likert Scale for some constructs, and a 5 point Likert scale for other constructs will be suitable when developing a questionnaire? I have 7 constructs ...
4
votes
1answer
360 views

Internal consistency reliability in item response theory models

How can internal consistency reliability of a test and of individual test items be quantified in Item Response Theory models? I know I can resort to Classical Test Theory, Cronbach's alpha, and other ...
4
votes
1answer
2k views

Item analysis for a likert-type questionnaire - item discrimination, point-biserial, factor analysis, cronbach's alpha, other diagnostics?

I'm working on a questionnaire that is intended to measure two constructs. There are presently 26 items in the questionnaire -- eleven for construct 1 and 15 for construct 2 -- and each item has five ...
4
votes
2answers
395 views

When do you consider a variable is a latent variable?

The problem is to define when a variable might be considered as a latent variable. I am interested in how to describe a latent variable, and what are the properties of latent variables. My twofold ...
4
votes
2answers
201 views

What is the reasoning behind the formulae for the different standard errors of measurement?

McManus (2012) discusses the three standard errors of measurement and provides the following definitions. $SE_{meas} = SD \sqrt{(1 - reliability)}$ The standard error of measurement is an estimate ...
4
votes
2answers
71 views

How to deal with the cases when participants gave rating of 2.5 or 4.5 on a 1-5 scale?

Because this was done during the interview, the participants just spoke out their rating rather than marked it done on the paper: they sometimes say "I think it is between 2 and 3" or "I give it 4 or ...
3
votes
3answers
1k views

A multitrait-multimethod matrix and data set

I am working my way through Multitrait-Multimethod Matrix in a psychometrics class. We're only required to be able to analyze them but I'd really like to be able to construct them. I think I am able ...