# Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

562 questions
Filter by
Sorted by
Tagged with
223 views

### Inter-rater reliability for "time-series"

I would like to calculate the inter-rater reliability between 3 raters for some kind of time-series. Actually, I'm not quite sure, if I can call my data time-series but I will give you an example, so ...
223 views

### How can I use the ICC as a measure for inter-rater agreement on a set of (more or less) correlated items?

I have a question about inter-rater agreement using the ICC. I'm trying to test, whether my study participants (the raters) rate the personality of a fictional character similarly - think of a ...
5k views

### Reliability testing for questionnaire with nominal multiple choice questions (one possible answer)

I have used a questionnaire to collect data from 200 people. This questionnaire was obtained from a published research paper in which researchers had used it in a specific population. It involves 34 ...
49 views

### Source to explain my bosses basic things about statistics

I'm working for a global multi-billion dollar company with a safety-relevant machine. During development of that machinery product we have a strategy to test for machine reliability. I had to review ...
1 vote
16 views

### RM Anova or 2 paired t-tests ? And verifying for individual differences with RCI

For a research project evaluating the changes in cognitive abilities after anesthesia using tests that record the reaction time to a stimulus, we tested our subjects on baseline, 1 hour and 4 hours ...
40 views

### Confusion about Standard Error of Measurement (SEM)

The problem is this: I want to "estimate the error made when attempting to predict the true score from the observed score when I am using an unreliable measurement" (Theory of Mental Tests, ...
1k views

### Inter rater reliability with multiple raters, multiple categorical variables with different levels

Apologies if the answer to this question exists elsewhere, but I haven't found anything quite matching what I'm looking for. I have a dataset wherein 3 raters independently assessed 10 subjects by ...
301 views

### How to calculate composite score with a 2-factor scale

Suppose I have a scale with four items meant to measure, say, happiness. I ran a confirmatory factor analysis and the model fits well with two factors and let's say that the two factors are 1) ...
251 views

### ANOVA with unreliable measure

I’m doing some literature review on sports science, in particular the effect of training on endurance. There is no standard endurance tests. Some consist in intermittent contractions until exhaustion ...
2k views

### Implementation of Krippendorff's α for unitizing continuous data

Krippendorff has a paper (free and official versions) describing an adaptation of his measure of reliability α to continuous data, such as text. Specifically, it calculates the reliability of both ...
2k views

### How to calculate two-way mixed ICC?

I want to calculate ICC for two-way mixed model, single rater, and either for consistency and absolute agreement. I found two packages for calculating ICC (intra-class correlation coefficient), that ...
158 views

### Question on the effect of test reliability on weighting of a test battery

Originally, a test battery had 4 components: two 100 item multiple-choice tests, one oral test, and one essay test. Each component measured different topics. Each of the 4 components was weighted 25%. ...
1 vote
24 views

1 vote
95 views

### Cronbach's Alpha of Likert Item

My questionnaire consists of ten 5-point scaled Likert items. If the Likert items have a good Cronbach alpha, is it ok to take the average score of 10 items?
223 views

### Computing Cronbach’s Alpha for Test with Random Items

I need to find out Cronbach's Alpha (CA) for a test that pulls 10 questions randomly from a pool of 40 questions for every subject that takes the test. As a result, almost every test has a different ...
1 vote
196 views

### Cronbach alpha and item selection

I am a relative novice when it comes to statistical analysis, so forgive me if my question is unclear, or simply stupid. I am also not a mathematician, as I work in social science research, but I have ...
29 views

### Reliability of the scale

Do we have to calculate Cronbach's alpha for all the questions ( items) or just the variables? I have 7 variables: 4 independent, 1 dependent, 1 mediating, and 1 moderating, and 23 items 23. ...
1k views

### Cronbach's alpha for one scale but different types of items

I developed a questionnaire on art expertise. There are 3 rather subjective questions("I know much about art","I know more than the average citizen","I am able to identify an ...
1 vote
193 views

### Internal consistency of correlation [duplicate]

I have this question below , and I am unable to understand what is internal consistency, can anyone please tell the concept , I have read its wiki page but I couldn't understand how to solve a ...
1 vote
216 views

### Low internal consistency

I am currently analyzing the data for a survey I have completed. In doing so, one question is quite simple (nominal) and require a typical “I do/I do not” and “I do not/I do” response from the ...
1 vote
584 views

### Measures of internal consistency of survey items

I have a question on a survey that asks students how useful a feature of the software was that they used for learning. I then have three questions that assess in what way the feature was useful. The ...
199 views

### What is the meaning of "internal consistency is 96% alpha"?

Gilliam Autism Rating Scale (GARS-1) GARS Scale is a checklist that helps identify autistic individuals. Reliability of internal consistency of this scale is 96% alpha. Part of a psychological ...
20 views

### How to format data for one-way intraclass correlation coefficient?

I'm testing inter-rater reliability of an instrument on a sample of subjects from several sites. Each subject is assessed by two raters, different between each site. According to this paper, I should ...