A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki).

learn more… | top users | synonyms

5
votes
0answers
24 views

Item reliability or fit in IRT-based adaptive tests?

My background is in machine learning and statistics, but I am relatively new to psychometrics and testing. Nearly all of the literature I've found on item reliability refers to Chronbach's alpha or ...
1
vote
0answers
10 views

can the same item be used in multiple constructs?

I am constructing a survey with around 10 multi-item scales. My piloting indicates that one of the most important multi-item scales (COP) has very poor reliability (.49), but there is an item in ...
4
votes
1answer
16 views

How to compare Cronbach's alpha when the grouping variable is continuous?

I want to test if the $\alpha$ for a scale is dependent upon a personality test, which is a continuous variable. I understand that there are many methods to test if two $\alpha$'s are significantly ...
0
votes
0answers
21 views

Replication reliability analysis

does anyone know about replication reliability analysis? I'm conducting a research which is similar to the study below (measure inter- and intra-subject reliability): ...
0
votes
0answers
29 views

Inter- and intra- reliability (intraclass correlation)

I am having some troubles in analyzing my research data. I would like to measure the inter- and intra-subject reliability in preparing three different samples (i.e. mild, moderate & extreme). Each ...
0
votes
1answer
29 views

Shall I remove factors because of low Cronbach alpha level?

I've got 2 questions actually. I am adopting a scale from USA to Turkey; the original scale has six factors, but based on the results of factor analysis, I found that adopted one consists of 9 ...
0
votes
0answers
11 views

Statistical formula for intra-rater reliability

I designed an assessment rubric for a class assignment and wish to calculate my own intra-rater reliability. My goal is to use the rubric to evaluate the same ordinal data at two different points in ...
3
votes
0answers
33 views

How to measure the reliability of a consensus ranking (problem from Kemeny-Snell book)

Suppose that $k$ experts are each asked to rank a set of $n$ objects in order or preference. Let allow ties in the rankings. John Kemeny and Laurie Snell in their 1962 year book "Mathematical models ...
0
votes
0answers
8 views

Starting from available data, how do I test the reliability of a technique that generates a score?

I have hundreds of samples of papilloma which are caused by human papilloma virus 6 or 11. There are many ways to determine which HPV is in a specimen. Is it 6 or 11 or a combination of both? We chose ...
0
votes
0answers
27 views

Sample size calculation to demonstrate reliability of physical test method

I'm preparing a procedure for validating attribute physical test methods (e.g. vision system) by demonstrating the reliability of the test method to detect known good/bad samples.I have sourced the ...
10
votes
2answers
187 views

Identification of useless questions from a questionnaire

I'm developing a questionnaire. To improve its reliability and validity I want to use statistical methods. I want to eliminate questions whose answers are always the same. This means that nearly all ...
0
votes
2answers
308 views

Reliability of a network

Consider the communication network shown in the figure below and suppose that each link can fail with probability p. Assume that failures of different links are independent: All links ...
1
vote
0answers
28 views

Repairable system and the sum of GEV random variables

We know that $X\sim {\mathrm {GEV}}(\alpha ,\beta ,0)$ and $Y\sim {\mathrm {GEV}}(\alpha ,\beta ,0)$ then $X+Y\sim {\mathrm {Logistic}}(2\alpha ,\beta )$. I am wondering, what will be $X+Y+Z$ ...
4
votes
4answers
145 views

If a measure is valid (but not necesarily reliable), can it be consistently replicated?

I had this question on an exam, and I was positive that the answer was A. The Beck Depression Inventory is a scale intended to measure depression levels, with higher scores indicative of higher ...
0
votes
1answer
73 views

Weibull analysis [duplicate]

I need to get the maximum likelihood estimate of a two-parameter Weibull distribution. I have tried it on my own but everything tends towards using a Newton-Raphson iterative method which I am not ...
0
votes
0answers
15 views

How can we measure between-experimenter reliability for experimental data?

Description of my situation: I will run an experiment in a few weeks with a total sample size of N = 70. Ideally I would prefer to have only one experimenter to collect all the data but I have to do ...
0
votes
0answers
17 views

Drawing an ROC curve from one point

First, thanks for the wonderful site. Second, I am trying to use an ROC curves to check the reliability of some nondestructive testing of concrete slabs. As you may know, the Non-destructive ...
1
vote
0answers
28 views

Optimising test re-test reliability through randomly generating hypothetical repeated trials

I have had a look around the forum, and despite some similar questions on measurement error, no one appears to have asked this question specifically. I have developed a test where a small group of ...
0
votes
0answers
47 views

Assessing questionnaire reliability using test-retest measures

I am an undergraduate student with not much knowledge regarding statistics. I was hoping you would be able to help me by answering a question. I am currently checking the reliability and validity of ...
2
votes
1answer
26 views

How to create an “alignment” variable from mutual rating variables

I want to create an "alignment" variable out of two other variables. Meaning, that if two raters (say manager and employee) rate each other similarly, alignment is high, but if they rate each other ...
0
votes
2answers
76 views

Do reliability and construct validity have same technical(statistical) assumptions?

It appears that reliability and construct validity have a similar backing in terms of statistics. Is it true ?
1
vote
1answer
19 views

Reliability measure with no single item scores

Let's say, I have an intelligence test made of 60 questions (items) which produces results for two intelligence constructs on a 0-100 scale (for instance crystallised and fluid intelligence). I only ...
-1
votes
2answers
116 views

What is the relation or difference between reliability and validity?

A test should be designed in a manner to get information to be believed in by investigator and academia world .
3
votes
1answer
116 views

How does reliability and validity affect the results (descriptive statistics)?

Okay, I think this might be a stupid question but I just can't quite seem to get there. I understand what reliability and validity is, but I don't understand just what I should do with it when looking ...
0
votes
0answers
20 views

Inter-coder reliability for continuous data and 4 coders

I've had four students coding the frequency of certain verbal and nonverbal behaviours. In hoping to generate some sort of intercoder reliability, I've had them all code the same 10 interviews. For ...
0
votes
0answers
26 views

Does true correlation in statistics represent validity coefficent as used in psychometrics? [duplicate]

The statistics appears to have borrowed certain concepts and models from psychometric theories and models. Is it a correct to presume that the two terms are equivalent?
1
vote
2answers
43 views

Confusion relating to Alpha Cronbach

I am still confused in using Cronbach's Alpha. Do we have to use it in sample survey before going for final survey? or is value of Alpha calculated based on the final survey data? Similarly, what does ...
1
vote
0answers
144 views

Intra- and inter- observer error, and the intra-class correlation coefficient (ICC)

I have 3 observers, who each perform multiple observations in which they count a phenomenon. I'm trying to quantify both inter- and intra- observer error for this study. I'm new to this and am finding ...
4
votes
1answer
201 views

Good internal factor structure but poor cronbach alpha?

I am running a CFA and getting good fit indices (CFI = .99, RMSEA = .01) for a uni-dimensional scale. However, when I test for internal consistency, I get poor Cronbach alphas (alpha = .6). I have ...
3
votes
1answer
339 views

Cronbach's Alpha with missing data

In R, there are several packages that calculate Cronbach's alpha. If the data contain no missing values, all packages converge to the same value. With missing data, ...
1
vote
0answers
117 views

How to analyze intra-rater reliability with 31 raters who all rated 4 subjects twice

I'm trying to analyze the intra-rater reliability of an occupational therapy assessment. The assessment consists of 57 items, where some items are ratio data and some items are ordinal data. In my ...
1
vote
1answer
250 views

How do you derive the Success-Run Theorem from the traditional form of Bayes Theorem?

In my industry it is common to test a sample of 20-30 and then use that data to draw conclusions about the reliability of the product with a certain confidence. We have tables for such things but it ...
1
vote
0answers
54 views

Is Cronbach's alpha interchangeable with $\rho$?

I am trying to calculate the reliability of my change scores using the following formula. Is $r$ ($\rho$) in this instance interchangeable with Cronbach's alpha? $$ \frac{.5(r_{xx}+ r_{yy}) - ...
1
vote
1answer
241 views

Cronbach's alpha vs. KR20/KR21 in reliability studies

Should we calculate Cronbach's alpha instead of KR20/KR21 for measuring the reliability of an achievement test, or a test with dichotomous data?
-1
votes
3answers
156 views

Minimum items for questionnaire

Is there a minimum item for questionnaire? Some researchers said that the total items for teenagers ranging from 80-110 items
9
votes
2answers
759 views

Where do the descriptors for Cronbach's alpha values come from (e.g., poor, excellent)?

It seems fairly common to describe Cronbach's alpha values as follows: α ≥ 0.9 Excellent 0.7 ≤ α < 0.9 Good 0.6 ≤ α < 0.7 Acceptable 0.5 ≤ α < 0.6 Poor α < 0.5 Unacceptable Where ...
0
votes
0answers
183 views

Sample size for multiple rater inter-rater reliability study

How do I calculate the minimum overall sample size required for a study using multiple raters (n=5) assessing inter-rater reliability using Fleiss generalized kappa for binary variables, Kendall's W ...
3
votes
1answer
183 views

What is the correct use of Cronbach's alpha?

I am trying to use Cronbach's alpha in order to see if some variables are measuring the "same", but I have some doubts about the correct use: Can I compute alpha for numerical, ordinal or ...
0
votes
0answers
47 views

Implementation of Krippendorff's α for unitizing continuous data

Krippendorff has a paper (free and official versions) describing an adaptation of his measure of reliability α to continuous data, such as text. Specifically, it calculates the reliability of both ...
1
vote
1answer
69 views

Results of bootstrap reliable (hyperbolic distribution)?

I already asked a question about the interpretation of the results of the bootstrap algorithm in case of a normal mixture. This time, I fit a hyperbolic distribution. According to this thread: the ...
2
votes
1answer
91 views

Determining which scale items to use for further analysis

From a questionnaire survey I have 44 statements (items) which were supposed to measure 11 bases of power (factors). Four items per factor. Since this survey was only slightly altered from its well ...
13
votes
4answers
247 views

What are the case studies in public health policy research where unreliable/confounded/invalid studies or models were misused?

I am drafting a literature review on a current public health issue where data are confounded: What are common historical case-studies that are used in public health/epidemiology education where ...
3
votes
2answers
265 views

Results of bootstrap reliable?

I am using the bootstrap algorithm to compute standard errors of the estimates of my normalmixEM output. I am not really sure if they are reliable? My code is (data here): ...
1
vote
1answer
55 views

Controlling for baseline imbalance

I have a quick question about growth curve modelling and baseline imbalance: do growth curve models control for baseline imbalance and if so, do they control for imbalance in a reliable way, i.e., ...
4
votes
3answers
108 views

Which test to see if two set of continuous data are comparable

Scenario: Two different persons have measured lesions in millimeters before and after medicine, so the data are continuous. I have calculated both of these readers' difference in measurements and ...
3
votes
1answer
93 views

Reliability engineering: Measuring the proportion of defective products at several time points using the same test units

A product being designed will have a failure-free warranty requirement of 1 year and the Engineers are planning a series of tests to estimate our warranty exposure. Based on the binomial confidence ...
1
vote
1answer
102 views

Two treatments - How do I compute reliability estimates like Cronbach $\alpha$?

I randomized my survey, and now I have my data on SPSS (and all cleaned up) but when trying to run my Cronbach's alpha I have a problem. Because I have two treatments (sexualised and normal) all my ...
5
votes
1answer
154 views

How can reliability and validity of content analysis be quantified when there is only one person coding the data?

How can reliability and validity of content analysis be quantified when there is only one person coding the data?
2
votes
0answers
110 views

Comparing two dependent intra-class correlations (ICCs)

We are measuring two waves from a biological signal. As an example consider the ECG $P$ wave and $T$ wave. From a single trace, the amplitude of both of these waves is measured. We want to compare ...
1
vote
1answer
134 views

Using existing questionnaire

I will be doing my colloquium and will be using existing questionnaires. I would like to know if there are authors/authorities who say that an existing questionnaire need not be validated. The ...