Questions tagged [psychometrics]

Psychometrics has evolved as a subfield of psychology to become the science of measurement of unobservable individual characteristics.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

psych package: reliability, should complete cases be used?

Would like to seek advice on instrument validation and the psych package in r. I would like to evaluate the reliability first (want to replicate the study here: https://doi.org/10.1186/s12885-018-5135-...
user avatar
  • 165
0 votes
1 answer
23 views

Factor analysis with multiple correct responses to questions

I have a data relating to a questionnaire that takes form of a knowledge test, in which each question has multiple correct responses and the respondents can check all responses they consider to be ...
user avatar
  • 1
1 vote
0 answers
16 views

Which intraclass correlation coefficient (ICC) to use to verify the test-retest reliability of a questionnaire?

I want to check the test-retest reliability of a questionnaire. -A total of 300 adolescents from the same school (10 classes) participated in the study. -The instrument was applied in two moments with ...
user avatar
1 vote
1 answer
36 views

Analysing multiple-choice question with multiple answers

I recently ran an experiment with 4 conditions where participants got a multiple choice task to select words best describing a subjective sensation. Now I'm struggling to figure out which analytical ...
user avatar
  • 11
0 votes
0 answers
78 views

Is this documentation for the R 'psych' package ICC() function incorrect?

I am not a statistician, but from my reading about various types of intraclass correlation measures, ICC(2,1) (Shrout and Fleiss nomenclature) is a measure of absolute agreement sensitive to additive ...
user avatar
  • 145
2 votes
1 answer
37 views

What are typical approaches to express the reliability of a categorical scale?

I have categorical scales consisting of items with each three answer options: 1) very much; 2) somewhat; 3) never. What would be a suitable approach to express reliabilities for scales like this? I am ...
user avatar
  • 469
0 votes
0 answers
7 views

How to compare two personality instruments (short versions vs longer version)

I need to compare two personality instruments (find correlations). One is a short version (less items) and the other is a long version (more items). The items (sentences) in short version are the same ...
user avatar
  • 1
1 vote
1 answer
28 views

Psychometric test validation question

We are trying to validate a test for young children using IRT analyses (rasch/2pl). Models do not converge due to a floor effect. Teachers also took this assessment. Would it skew results if i ...
user avatar
  • 11
0 votes
1 answer
61 views

Multiple Regression: Why is there range restriction in predicted values?

I have a built linear regression model whereby I have used scores on a personality survey to predict manager ratings of job performance. Both my predictor and outcome variables have a min and max ...
user avatar
  • 3
0 votes
0 answers
52 views

When to use "fixed number of components" in a PCA

The question is relatively simple (and stupid maybe). When do I use a fixed number of components (especially in analyzing a questionnaire). Is it useful when I have literature to a validated scale/...
user avatar
1 vote
0 answers
105 views

Using Pearson Moment Correlation For Establishing Validity Between Two Questionnaires

I'm moderately new to psychometrics and test design, and I'm currently in the process of attempting to develop a questionnaire that measures soft skills (communication, collaboration, work ethic, ...
user avatar
  • 11
1 vote
2 answers
49 views

I want to test for a significant difference in the result (t-value or effect size) of two paired t-tests

I am testing the quality of a measurement construct, where I propose that Measure A is better (i.e. more responsive to situational changes) than Measure B is. I conduct a study in which I measure the ...
user avatar
  • 11
0 votes
0 answers
27 views

Gwet's AC for single items and summated scale scores: nominal, ordinal or continuous data?

I really hope someone can help me. I am measuring the test-retest reliability of a scale and I am unclear of what kind of variables I am dealing with. I have looked at the following threads: What is ...
user avatar
  • 11
0 votes
0 answers
68 views

lavaan factor scores to feed other models

Let's suppose I have an outcome variable that is not continuous. For example, it could be an ordered categorical variable or a nominal (unordered) categorical variable. This implies that I would need ...
user avatar
0 votes
0 answers
19 views

IRT models for repetitive items

I'm looking for literature about IRT models that fit for tests in which items are presented for each examinee more than once and/or different questions are asked for the same item. For example, in the ...
user avatar
  • 1
0 votes
0 answers
44 views

Valid 2nd order SEM with multiple dependent variables

I'm working on an SEM problem, and I could use some second opinions! The diagram below shows the proposed SEM model, where I have a 4 latent constructs which are estimated themselves from 2 second-...
user avatar
0 votes
1 answer
54 views

Showing that estimating $Y$ from $X$ via linear regression and estimating $X$ from $Y$ via linear regression are not invertible

This is a follow up to the question Regression on population values?. In the theory of test equating (see, e.g., Test Equating, Scaling, and Linking by Kolen and Brennan (2014)), suppose we have a ...
user avatar
  • 3,883
4 votes
2 answers
74 views

Regression on population values?

In the theory of test equating (see, e.g., Test Equating, Scaling, and Linking by Kolen and Brennan (2014)), suppose we have a group of examinees whom we randomly assign one of two exam forms, say ...
user avatar
  • 3,883
2 votes
1 answer
126 views

What is the minimum acceptable correlation for creating a composite variable?

I have created a composite measure comprising two items (both 7-point Likert scale items), using the summed z score method. The correlation coefficient for these two items is 0.59. This seemed ...
user avatar
  • 21
1 vote
1 answer
37 views

Assess overall question difficulty in Graded Response Model

I used graded response modelling (GRM) to assess the difficulty of items from a survey using a Likert scale from 1 to 5. The results show the difficulty of each item (i.e., 1-5), for each question. I ...
user avatar
  • 11
1 vote
0 answers
17 views

Denotion of Cronbach's Alpha based on Polychorric Correlations

we assessed several 5 point Likert-Scale items in a questionnaire and are interested in Cronbach's $\alpha$ for the internal consistency. I recently read (here) that it can be more precise to used ...
user avatar
  • 61
4 votes
2 answers
76 views

When and why should psychologists use ordered probit models rather than the general linear model?

Psychologists often use the general linear model with ordinal independent/dependent variables (i.e. Likert scales to measure 'levels' of a psychological trait. For example, assigning numbers to the ...
user avatar
1 vote
0 answers
90 views

How to assess agreement among measurement models (factor analysis) from multiple studies?

For a meta-analysis, I've been collating studies that apply factor analysis on likert-like scale surveys for a specific psychometric construct. The original speculation from source material is that ...
user avatar
  • 3,486
2 votes
1 answer
158 views

Inclusion of a NA category in inter-rater agreement calculation

I have a dataset in which the behaviour of $n = 6$ subjects have been rated by $k = 84$ raters on an ordinal scale with 5 ordinal categories. The distances between the ordinal categories are not ...
user avatar
  • 75
3 votes
1 answer
52 views

How can I compare the variance or correlation between ranked *single answer* polls?

I am an undergrad psych major examining the differences in personal preferences between the extremist website 4chan and other websites, which will represent the pop norm. I would like to test whether ...
user avatar
1 vote
1 answer
76 views

Bayesian Multiple Item Response Models?

I am trying to extend the ideas of item response theory to multiple responses. Consider a marketing survey, which asks customers, "what's the deciding factor in whether or not you purchase ...
user avatar
  • 1,700
6 votes
1 answer
67 views

What is the role of non-anchor items when equating scales across studies using IRT?

I'm trying to use IRT to equate scales across 5 datasets. Three of them use the same scale (A); the other ones use different ones (B and C) There are items that overlap between A and C, A and B, but ...
user avatar
  • 331
2 votes
0 answers
76 views

Are there supposed to be bounds on parameters in 2PL Item Response Theory models?

Recently I've been studying Item Response Theory (IRT) and have come across some issues with the application side of it. I currently have a dataset of ~200 respondents x 7405 questions (quite ...
user avatar
4 votes
1 answer
200 views

Do 1PL IRT models measure both ability and difficulty, or just difficulty?

I'm trying to better understand Item response Theory (IRT) from a Bayesian perspective. Hypothetically, suppose I want to use a 1PL model and my data is a binary matrix ...
user avatar
  • 1,700
0 votes
2 answers
70 views

Is this an instance of convergent evidence?

In psychometrics, convergent validity is a source of proof for construct validity. It is said that it is the degree to which test scores are correlated with test scores of relevant constructs or the ...
user avatar
  • 1
0 votes
1 answer
180 views

IRT: What is enough information? And can a test provide too much?

I was wondering what makes for a good test information value in IRT, and came across this: Reliability: Item and test information functions graphically reflect how reliably the individual items and ...
user avatar
  • 327
1 vote
0 answers
21 views

Non-independence of trial likelihoods in a staircase procedure?

In psychometrics, we often want to know, for instance, a given participants' perceptual threshold: the intensity of a stimulus that they can detect 50% of the time. It's common to use staircase ...
user avatar
  • 4,881
0 votes
1 answer
220 views

Calculating Theta scores for identified subjects (using R package ltm) [closed]

I am using the R package ltm to calculate a 2PL IRT model. goal: creating a list of subjects IDs with the calculated Theta scores. problem: ltm requires that the input includes only item scores, and ...
user avatar
  • 13
2 votes
1 answer
650 views

What is the difference between Factor Analysis and Path Analysis in Structural Equation Modelling?

I am still a beginner in Structural Equation Modelling and I don't have a solid background in Statistics, so I apologize if my question is a bit naïve. What I traditionally do when creating a ...
user avatar
7 votes
1 answer
614 views

Analyzing a partially crossed design

I have a data set from a test (see below). The scoring algorithm gives each item (item_id) a score (y) that is continuous from $...
user avatar
3 votes
0 answers
161 views

Does it make sense to use the Spearman-Brown correction (prediction / prophecy) formula with Kendall's Tau correlation coefficient?

The Spearman-Brown prophecy formula is used to estimate the increase in reliability that is obtained when more measurements are taken (and aggregated) from the same construct. Reliability is the ...
user avatar
  • 14.5k
1 vote
0 answers
83 views

How to use Cronbach's Alpha?

Is it appropriate to use Cronbach's Alpha for nominal variables? I have 20 nominal variables (two dimensions with 10 questions each) that ask about the characteristics of work performance. The ...
user avatar
3 votes
0 answers
25 views

How to deal with data from different centers?

I want to examine the psychometric properties (e.g. internal consistency, factorial validity) of a scale. The data (N=700) comes from 16 different locations. Central means and variances did not differ ...
user avatar
1 vote
0 answers
62 views

Performing a CMH test in R [closed]

I'm trying to perform a CMH test to examine differential item functioning by sex in R. I've tried cmh.test and mantelhaen.test ...
user avatar
1 vote
1 answer
83 views

Why can exploratory factor analysis (EFA) suggest a single factor, yet a 1-factor confirmatory factor analysis (CFA) model not fit?

I know I previously saw a thread on this topic, but I cannot locate it. So, please forgive my redundancy. I have an 8-item questionnaire administered to a (small) sample of 59 patients. The data ...
user avatar
0 votes
1 answer
25 views

Can I have 2 Cronbach's alphas?

I've got an overall Cronbach's alpha of -.3. The questions are reverse scored. When I calculate 5 out of the 10 questions which have high to low scoring, alpha is .8, and when I find alpha for the ...
user avatar
  • 1
1 vote
1 answer
62 views

Inferential interpretation of Kappa reliability

I have obtained a Fliess' Kappa interrater reliability of $37.5\%$ from $90$ randomly selected raters rating $100$ randomly selected, binary items ($Yes/No$). Question: Is the following ...
user avatar
  • 3,136
2 votes
1 answer
313 views

How to intrepret one significant and one insignificant mediator and a significant direct effect in PROCESS?

I have conducted a mediation analysis with two mediators using PROCESS. My findings are that in X -- a --> M1 -- b --> Y, a and b are significant. However in X -- c --> M2 -- d --> Y, only d is ...
user avatar
  • 21
2 votes
0 answers
407 views

How to test for unidimensionality using the mirt r-package?

As it is my understanding that the ltm-package is obsolete, I'm looking for an equivalent to ltm::unidimTest() in the mirt-...
user avatar
1 vote
0 answers
36 views

Are there any simple pairwise comparison tests suitable for comparing emotions?

So, I'm pretty new to using statistics in a research context. I am currently writing my thesis in didactics of mathematics. In my study, a group of $10$ students are to solve two different ...
user avatar
1 vote
0 answers
148 views

Cronbach's Alpha with repeated missing values

I would like to use Cronbach's Alpha to check the fiability or internal consistency of a cognitive task. The task has a certain number of trials which were coded with zeros (incorrect trial) and ones (...
user avatar
0 votes
1 answer
98 views

Is signal detection theory a Bayesian approach?

Signal detection theory can be applied to modeling perception thresholds for defined stimuli, for example the detection of the presence of a specific signal. It is assumed both the stimulus intensity ...
user avatar
0 votes
0 answers
23 views

Relationship between people's insight and behavior

My main interest is to see how well people can identify the reasons behind their decisions. To measure this, I have collected 2 types of data: one set of variables that measures how important people ...
user avatar
  • 3
0 votes
0 answers
25 views

Generalizability study of subscaled rating instrument

I have a school rating ($1$ thru $4$) instrument that consists of $9$ subscales (e.g., classroom instruction, school management etc.). Under each subscale, I have $6$ items on which rating occurs. I ...
user avatar
  • 3,136
0 votes
1 answer
258 views

lmer fixed effects coefficients: which is the term of comparison for levels? [closed]

I have a lmer model with two within factors lmer (variable ~ time * condition + (1|id), data= df) with time having 2 levels (<...
user avatar
  • 1

1
2 3 4 5
7