A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki).

learn more… | top users | synonyms

0
votes
1answer
11 views

When is a poor reliability index acceptable?

I have tested this particular scale and it turns out that it has low reliability index (a=4.5). I have already deleted two items but it only increased a bit by a=.67. My sample size is 25 as part of ...
1
vote
0answers
18 views

which statistic should I use for inter-rater agreement on nominal data, multiple responses

I have over 400 open-ended responses. The total has been coded by one rater, and a 10% identical sample of the total has been coded by 2 raters. There is a choice of 14 nominal codes, of which the ...
1
vote
1answer
18 views

Cronbach's Alpha low- significant results

One of my variables has a low Cronbach's Alpha. I continued to run an planned contrast to test my hypothesis and the results were significant. I understand that the results should be interpreted ...
0
votes
0answers
11 views

Effect of Violating Spearman_Brown Prophecy Formula Assumptions

The Spearman-Brown Prophecy formula is sometimes used to predict the reliability of the average assessment based on N raters using the interrater reliability calculated from the N sets of individual ...
0
votes
0answers
8 views

Reliability Measure for Unequal Subjects between Raters?

I am comparing two medical imaging analysis techniques, and I would like to assess the inter- and intra-rater reliability for using each method. I am planning the study design, and would like help ...
2
votes
0answers
73 views

Variance component analysis with lme4

I am beginner with lme4 and I am seeking some advice on how to carry a variance component analysis. The data come from a real-world scenario, not from a designed experiment, resulting in a complex ...
3
votes
0answers
57 views

What is the relationship between scale reliability measures (Cronbach's alpha etc.) and component/factor loadings?

Let's say I have a dataset with scores on a bunch of questionnaire items, which are theoretically comprised of a smaller number of scales, like in psychology research. I know a common approach here ...
0
votes
1answer
61 views

Can ICC values be negative?

I am calculating the ICC as per the method of Shrout-Fleiss (1979), using the (3, 1) model in which the judges are fixed as are the targets. Some of my ICC values are negative. Is this to be ...
0
votes
1answer
38 views

Measuring Cronbach Alpha for very large number of items

How can I calculate Cronbach Alpha for two tests that I have administered to 165 people? One test contains 110 items and the other contains 67 items. The scoring is done on the hard copies and the ...
3
votes
1answer
52 views

Why change items which don't change Cronbach Alpha?

I am doing peer review on a paper submission for a journal, and I am somewhat out of my depth. The authors are developing a psychometric scale. They did a pilot study with 45 items. They measured ...
7
votes
2answers
325 views

How is the Spearman-Brown prophecy formula affected by questions of differing difficulties?

How are the results of the Spearman-Brown prophecy formula affected by having test questions of differing difficulties or raters who are easy or hard graders. One respected text says the S-B is ...
0
votes
0answers
3 views

Reliability estimate for each item (comparing raters to each other)

I am working with a 15 item scale of pre-school teaching. Five observers observed 58 teachers. Its a complete crossed matrix i.e, all raters observed all teachers. In an attempt to weed out the weak ...
0
votes
0answers
5 views

Assessing test-retest reliability of a questionnaire [duplicate]

I am testing the test-reliability of a newly developed questionnaire over two time points. The study involves parents answering about their child over the two occasions. Would intraclass correlation ...
0
votes
0answers
12 views

Cronbach's Alpha for open questions

In the questionnaire I asked respondents from two countries how many job offers they received from 5 sources in the last 6 months. There are 5 questions - one for each source. It is an open question, ...
6
votes
1answer
83 views

Item reliability or fit in IRT-based adaptive tests?

My background is in machine learning and statistics, but I am relatively new to psychometrics and testing. Nearly all of the literature I've found on item reliability refers to Chronbach's alpha or ...
1
vote
0answers
19 views

can the same item be used in multiple constructs?

I am constructing a survey with around 10 multi-item scales. My piloting indicates that one of the most important multi-item scales (COP) has very poor reliability (.49), but there is an item in ...
4
votes
1answer
42 views

How to compare Cronbach's alpha when the grouping variable is continuous?

I want to test if the $\alpha$ for a scale is dependent upon a personality test, which is a continuous variable. I understand that there are many methods to test if two $\alpha$'s are significantly ...
0
votes
0answers
35 views

Replication reliability analysis

does anyone know about replication reliability analysis? I'm conducting a research which is similar to the study below (measure inter- and intra-subject reliability): ...
0
votes
0answers
66 views

Inter- and intra- reliability (intraclass correlation)

I am having some troubles in analyzing my research data. I would like to measure the inter- and intra-subject reliability in preparing three different samples (i.e. mild, moderate & extreme). Each ...
0
votes
1answer
106 views

Shall I remove factors because of low Cronbach alpha level?

I've got 2 questions actually. I am adopting a scale from USA to Turkey; the original scale has six factors, but based on the results of factor analysis, I found that adopted one consists of 9 ...
0
votes
0answers
30 views

Statistical formula for intra-rater reliability

I designed an assessment rubric for a class assignment and wish to calculate my own intra-rater reliability. My goal is to use the rubric to evaluate the same ordinal data at two different points in ...
3
votes
0answers
53 views

How to measure the reliability of a consensus ranking (problem from Kemeny-Snell book)

Suppose that $k$ experts are each asked to rank a set of $n$ objects in order or preference. Let allow ties in the rankings. John Kemeny and Laurie Snell in their 1962 year book "Mathematical models ...
0
votes
0answers
16 views

Starting from available data, how do I test the reliability of a technique that generates a score?

I have hundreds of samples of papilloma which are caused by human papilloma virus 6 or 11. There are many ways to determine which HPV is in a specimen. Is it 6 or 11 or a combination of both? We chose ...
0
votes
0answers
97 views

Sample size calculation to demonstrate reliability of physical test method

I'm preparing a procedure for validating attribute physical test methods (e.g. vision system) by demonstrating the reliability of the test method to detect known good/bad samples.I have sourced the ...
10
votes
2answers
222 views

Identification of useless questions from a questionnaire

I'm developing a questionnaire. To improve its reliability and validity I want to use statistical methods. I want to eliminate questions whose answers are always the same. This means that nearly all ...
0
votes
2answers
367 views

Reliability of a network

Consider the communication network shown in the figure below and suppose that each link can fail with probability p. Assume that failures of different links are independent: All links ...
2
votes
0answers
35 views

Repairable system and the sum of GEV random variables

We know that $X\sim {\mathrm {GEV}}(\alpha ,\beta ,0)$ and $Y\sim {\mathrm {GEV}}(\alpha ,\beta ,0)$ then $X+Y\sim {\mathrm {Logistic}}(2\alpha ,\beta )$. I am wondering, what will be $X+Y+Z$ ...
4
votes
4answers
261 views

If a measure is valid (but not necesarily reliable), can it be consistently replicated?

I had this question on an exam, and I was positive that the answer was A. The Beck Depression Inventory is a scale intended to measure depression levels, with higher scores indicative of higher ...
0
votes
1answer
90 views

Weibull analysis [duplicate]

I need to get the maximum likelihood estimate of a two-parameter Weibull distribution. I have tried it on my own but everything tends towards using a Newton-Raphson iterative method which I am not ...
0
votes
0answers
19 views

How can we measure between-experimenter reliability for experimental data?

Description of my situation: I will run an experiment in a few weeks with a total sample size of N = 70. Ideally I would prefer to have only one experimenter to collect all the data but I have to do ...
0
votes
0answers
24 views

Drawing an ROC curve from one point

First, thanks for the wonderful site. Second, I am trying to use an ROC curves to check the reliability of some nondestructive testing of concrete slabs. As you may know, the Non-destructive ...
1
vote
0answers
36 views

Optimising test re-test reliability through randomly generating hypothetical repeated trials

I have had a look around the forum, and despite some similar questions on measurement error, no one appears to have asked this question specifically. I have developed a test where a small group of ...
0
votes
0answers
84 views

Assessing questionnaire reliability using test-retest measures

I am an undergraduate student with not much knowledge regarding statistics. I was hoping you would be able to help me by answering a question. I am currently checking the reliability and validity of ...
2
votes
1answer
30 views

How to create an “alignment” variable from mutual rating variables

I want to create an "alignment" variable out of two other variables. Meaning, that if two raters (say manager and employee) rate each other similarly, alignment is high, but if they rate each other ...
0
votes
2answers
92 views

Do reliability and construct validity have same technical(statistical) assumptions?

It appears that reliability and construct validity have a similar backing in terms of statistics. Is it true ?
1
vote
1answer
28 views

Reliability measure with no single item scores

Let's say, I have an intelligence test made of 60 questions (items) which produces results for two intelligence constructs on a 0-100 scale (for instance crystallised and fluid intelligence). I only ...
-2
votes
2answers
137 views

What is the relation or difference between reliability and validity?

A test should be designed in a manner to get information to be believed in by investigator and academia world .
3
votes
2answers
163 views

How does reliability and validity affect the results (descriptive statistics)?

Okay, I think this might be a stupid question but I just can't quite seem to get there. I understand what reliability and validity is, but I don't understand just what I should do with it when looking ...
0
votes
0answers
30 views

Inter-coder reliability for continuous data and 4 coders

I've had four students coding the frequency of certain verbal and nonverbal behaviours. In hoping to generate some sort of intercoder reliability, I've had them all code the same 10 interviews. For ...
1
vote
2answers
56 views

Confusion relating to Alpha Cronbach

I am still confused in using Cronbach's Alpha. Do we have to use it in sample survey before going for final survey? or is value of Alpha calculated based on the final survey data? Similarly, what does ...
1
vote
0answers
209 views

Intra- and inter- observer error, and the intra-class correlation coefficient (ICC)

I have 3 observers, who each perform multiple observations in which they count a phenomenon. I'm trying to quantify both inter- and intra- observer error for this study. I'm new to this and am finding ...
5
votes
1answer
466 views

Good internal factor structure but poor cronbach alpha?

I am running a CFA and getting good fit indices (CFI = .99, RMSEA = .01) for a uni-dimensional scale. However, when I test for internal consistency, I get poor Cronbach alphas (alpha = .6). I have ...
3
votes
1answer
815 views

Cronbach's Alpha with missing data

In R, there are several packages that calculate Cronbach's alpha. If the data contain no missing values, all packages converge to the same value. With missing data, ...
1
vote
0answers
187 views

How to analyze intra-rater reliability with 31 raters who all rated 4 subjects twice

I'm trying to analyze the intra-rater reliability of an occupational therapy assessment. The assessment consists of 57 items, where some items are ratio data and some items are ordinal data. In my ...
1
vote
1answer
444 views

How do you derive the Success-Run Theorem from the traditional form of Bayes Theorem?

In my industry it is common to test a sample of 20-30 and then use that data to draw conclusions about the reliability of the product with a certain confidence. We have tables for such things but it ...
1
vote
0answers
72 views

Is Cronbach's alpha interchangeable with $\rho$?

I am trying to calculate the reliability of my change scores using the following formula. Is $r$ ($\rho$) in this instance interchangeable with Cronbach's alpha? $$ \frac{.5(r_{xx}+ r_{yy}) - ...
1
vote
1answer
324 views

Cronbach's alpha vs. KR20/KR21 in reliability studies

Should we calculate Cronbach's alpha instead of KR20/KR21 for measuring the reliability of an achievement test, or a test with dichotomous data?
-1
votes
3answers
207 views

Minimum items for questionnaire

Is there a minimum item for questionnaire? Some researchers said that the total items for teenagers ranging from 80-110 items
10
votes
2answers
2k views

Where do the descriptors for Cronbach's alpha values come from (e.g., poor, excellent)?

It seems fairly common to describe Cronbach's alpha values as follows: α ≥ 0.9 Excellent 0.7 ≤ α < 0.9 Good 0.6 ≤ α < 0.7 Acceptable 0.5 ≤ α < 0.6 Poor α < 0.5 Unacceptable Where ...
0
votes
0answers
188 views

Sample size for multiple rater inter-rater reliability study

How do I calculate the minimum overall sample size required for a study using multiple raters (n=5) assessing inter-rater reliability using Fleiss generalized kappa for binary variables, Kendall's W ...