Psychometrics has evolved as a subfield of psychology to become the science of measurement of unobservable individual characteristics.

learn more… | top users | synonyms

0
votes
0answers
2 views

How to calculate discrimination index and difficulty index of short answer test items?

How to calculate discrimination index and difficulty index of short answer test items? Should I use t test or the formula upper group score - lower group score divided by number of students in a ...
3
votes
1answer
215 views

Validation of a questionnaire in a new population

I have 400 responses to a 20 item questionnaire which purports to measure an attitudinal constuct in medical students. The instrument was validated in the US for a single year of medical students and ...
0
votes
1answer
67 views

Detecting Differential Item Functioning (DIF) using ordered logistic regression

I am in a situation in which I want to investigate whether test items of a questionnaire appear to have DIF. The items are polytomous and ranges from 1 to 5, with 5 being the most positive answer. ...
1
vote
0answers
17 views

Select and weigh questions on a reading assessment

I am a phd student in computer science and as such I am the goto guy for anything "mathy" in the cross discipline research group to which I belong. I have recently been given an assignment at work I'm ...
1
vote
0answers
5 views

Inter-raters reliability for sentiment analysis lexicon

I have a list of words (more than 2000) which were rated by 4 raters for sentiments like anger (scale from 1 to 5). I run an alpha test on SPSS which gives me a coefficient of >8 which shows a good ...
0
votes
0answers
5 views

What is the two alternative forced choice paradigm

Can someone please help me, Is a two alternative forced choice paradigm (2AFC) an experimental design?
1
vote
1answer
32 views

raising the probability of correct answers of nextItem (catR)

Using the catR package I am trying to create a nextItem call that respondents would answer correctly about 65% of the time (even if that means the method is not as efficient as it could be). Children ...
1
vote
1answer
653 views

Doing CFA on a known theoretical model, but having problems with convergent and discriminant validity

I did a survey based on the UTAUT model / theory with standard questions used. The N was 150. I am trying to validate the model for my survey with structural equation modeling (via the AMOS software). ...
4
votes
1answer
518 views

What is the relationship between scale reliability measures (Cronbach's alpha etc.) and component/factor loadings?

Let's say I have a dataset with scores on a bunch of questionnaire items, which are theoretically comprised of a smaller number of scales, like in psychology research. I know a common approach here ...
0
votes
0answers
14 views

Test-retest reliability

I have come across following table in a publication, where r is correlation coefficient. I could not find detailed procedures for test-retest reliability, is it any different from paired t-test ? ...
4
votes
1answer
268 views

Omega H asymptotic

In the omega function in the psych package for R, we have several outputs including Hierarchical omega, Asymptotic omega and Total omega. I was reading about the outputs, but I find nothing about ...
4
votes
4answers
369 views

Are discrimination parameters in two-parameter IRT models only specific to items?

Discrimination parameters in two-parameter model from IRT are usually considered item parameters. But I've come to doubt it. Think about psychophysics; for example, detecting luminance. I don't ...
0
votes
0answers
16 views

Calculate within and between group correlation

MY DATA: A surveys which contains 2 different questionnaires on the same subject which were taken by n=800 patients (i.e., timepoint) (#1 questionnaire: 5 questions which reach a total score of 0-42, ...
0
votes
0answers
20 views

Statistical Method for Normalization of Data

I collected data using a pre-test instrument to capture baseline emotions (two dimensions: valence and arousal) of participants. The two dimensions in the questionnaire range from calm to excited ...
3
votes
0answers
15 views

Comparing confirmatory factor analysis models with different number of observed variables

I have a 14 item scale that is hypothesized to measure self-esteem. The first model I specify uses all 14 items and predicts one latent factor (i.e., self-esteem). I would then like to specify a ...
4
votes
1answer
42 views

What is the difference between formative and reflective measurement models?

Can anyone provide examples as to why choose one over the other? Are they calculated diferently?
7
votes
4answers
15k views

What are the primary differences between z-scores and t-scores, and are they both considered standard scores?

We are currently converting student test scores in this manner : ( ScaledScore - ScaledScore Mean ) / StdDeviation ) * 15 + 100 I was referring to this ...
2
votes
3answers
189 views

“Two stage” factor Analysis: factoring saved factor scores

I have a model which consists of 19 questions, which are divided in three factors (factor 1 - nine questions, factor 2 - six questions, factor 3 - four questions). For this I did a factor analysis and ...
1
vote
0answers
16 views

CFA model with systematic temporal correlation between measures

I'm looking to fit CFA models to data in the case where measures have systematic temporal correlations between them. I'm familiar with autoregressive approaches, such as those described on Dave ...
3
votes
1answer
41 views

Non-Significant difference between individual items, but significant difference between overall scores

Say that we have a scale measuring Healthiness of Diet, and want to see if two populations have different levels of Healthiness of Diet. If we find that the difference between the two samples at the ...
3
votes
3answers
3k views

A multitrait-multimethod matrix and data set

I am working my way through Multitrait-Multimethod Matrix in a psychometrics class. We're only required to be able to analyze them but I'd really like to be able to construct them. I think I am able ...
7
votes
4answers
2k views

Validating an existing questionnaire into another language

I am doing research and trying to use a questionnaire that is available in English in another language. Is there a systematic approach for validating a questionnaire in another language? What ...
9
votes
3answers
222 views

Is it always better to extract more factors when they exist?

Unlike principal components analysis, the solutions to factor analysis models are not necessarily nested. That is, the loadings (for example) for the first factor won't necessarily be identical when ...
2
votes
1answer
54 views

Is empirical reliability in IRT the counterpart of internal consistency in CTT?

In IRT it is not meaningful to compute internal consistency coefficients such as Cronbachs alpha, because standard error of measurement (and thus measurement precision) is not assumed uniform across ...
2
votes
0answers
23 views

What measure of test reliability is appropriate if the final test output is not the sum of item scores?

I would like to compute a measure of reliability in the sense of Cronbach's alpha, McDonalds omega or glb. However it seems to me that all these measures assume that the final test score equates to ...
0
votes
0answers
49 views

What does 'downward pressure on sample size' mean?

In this article, there is this phrase 'downward pressure on sample size'. This is the paragraph: no matter the approach to collecting this new data, when attempting to ...
0
votes
0answers
40 views

Acceptable values for discrimination parameter - 2PL IRT Model

What is the acceptable range for discrimination parameters with the two-parameter IRT model? I have information from Baker (2001) but would love additional information. I have heard discrimination ...
1
vote
0answers
40 views

How can Bayesian inference improve upon logistic regression in incorporating psychometric data?

In order to estimate probability of default, banks and other financial institutions have often used logistic regression based on data involving credit history. Suppose an entrepreneur or borrower ...
0
votes
0answers
12 views

Combining different measures from different studies

I'm interested in combining outcomes from different psychiatric studies. However, the studies look at different types of diagnoses, that are measured sometimes with the same measurement, and sometimes ...
6
votes
2answers
2k views

Good internal factor structure but poor cronbach alpha?

I am running a CFA and getting good fit indices (CFI = .99, RMSEA = .01) for a uni-dimensional scale. However, when I test for internal consistency, I get poor Cronbach alphas (alpha = .6). I have ...
1
vote
1answer
375 views

Using Ltm package to calclate 3PL some lines of data seem to go missing

I am hoping someone can help with the following: I have been trying to use the ltm package in R to get person fit estimates using a 3PL model. The problem I have been having is that I start off ...
4
votes
1answer
91 views

What is Cronbach's Alpha intuitively?

I'm trying to understand Cronbach's Alpha intuitively. What is the general idea behind this construct? What properties were they trying to ensure that it had?
3
votes
0answers
25 views

Inter-rater agreement of a gold standard dataset - a ceiling for reliable evaluation of algorithms?

In my field, a dated gold standard dataset is used to track progress in algorithm development. Now when the state-of-the-art algorithms obtain higher correlation than is the inter-rater agreement of ...
1
vote
1answer
23 views

Can the test information function be used for a field test item bank?

I am field testing a few forms of items (with embedded anchor items) and will be analyzing the items using IRT, a 2PL model. These items will likely be used to create an item bank that can then create ...
1
vote
1answer
20k views

How to calculate validity of questionnaire in SPSS?

I have 3 types of questionnaires, A 15 item scale with each item on a 7-point Likert type scale. Six subscales each containing different number of items 21 subscales each containing 4 items. ...
0
votes
1answer
24 views

measure development - removing items from an item pool due to ceiling effect

During the early stages of scale development, many items within our item pool were found to have a strong ceiling effect with low score variability. In order to attempt create a meaningful measure ...
3
votes
1answer
62 views

The valid variance components to include in a generalizability theory model (anova) involving nesting? Are these correct interactions?

My question is about generalizability theory. A study by Bergeron, Floyd, McCormack, and Farmer (2008) is only adding to this confusion. Basically, in G-theory you want the first model you run to ...
1
vote
0answers
50 views

What do I do when my second order CFA model is not identified?

I am assessing a measurement model of a three-factor organisational commitment scale for my PhD project. Although congeneric single factor and first order CFA models have produced excellent model fit ...
4
votes
0answers
52 views

What's an intuitive way to explain the different types of validity?

Specifically I'm thinking of a simplified division whereby validity is divided into: Construct validity 1a. Convergent validity 1b. Discriminant validity Criterion related validity 2a. Predictive ...
4
votes
2answers
3k views

Can I combine several ordinal questions to create an index and/or composite measure?

I am using a survey that contains several questions about various dimensions of performance for policy research institutes. Here, performance in the policy arena is unpacked into things like: ...
0
votes
0answers
157 views

Interpreting summary of ICC results in R package “psych”

I have a dataframe "all.st.icc", where 27 subjects have measures of social behavior for each of 8 different years. First four rows of data are as follows: ...
0
votes
0answers
54 views

Is the use of PCA appropriate to validate a designed questionnaire?

I looked at the questions that may already have my answer but unfortunately it is not the case. I would like to ask again my question which has been revised to fit the rules. I am to measure 5 ...
0
votes
0answers
20 views

The differences between criterion validity and operational validity

In some psychological articles, I have seen the term "operational validity". It is often used instead of "criterion validity". What are the differences between these two concepts?
1
vote
1answer
34 views

Where I can find resources to learn how to calculate the sample size representativeness, and realiability and validity of questionnaires?

I'm totally newbie in psychometrics but starting from a research paper with full data I would like to understand: how to calculate if the sample size is representative; how to calculate reliability ...
1
vote
1answer
65 views

Questions regarding interpretation of item parameters: inconsistent results between classical item analysis and item response analysis

I was not trained as a psychometrician, so my questions may be concept-related. If you know different resources, books, or articles that can clear my confusions, please let me know. Thank you very ...
0
votes
0answers
62 views

Attitude measurement – Score classification – Why using percentiles?

Once we have raw scores from a group of Likert scale items, why are percentiles used instead of considering all the possible scores from the total number items when classifying results of individuals ...
2
votes
0answers
40 views

Combining different raters scores

In my experiment two raters evaluate 100 products, both expressing an "innovation score" on a likert scale (range is 1 to 5, where 1 is "not innovative" and 5 is "disruptive innovation"). I ...
2
votes
1answer
88 views

Item discrimination of poorly targeted items

In test theory, it appears to be widely recognized that very "easy" or "very difficult" items, relative to the ability of the sample of respondents, will have deflated correlations with the rest of ...
15
votes
4answers
9k views

How to get started with applying item response theory and what software to use?

Context I have been reading about item response theory, and I find it fascinating. I believe I understand the basics, but I am left wondering how to apply statistical techniques related to the area. ...