A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki).

learn more… | top users | synonyms

0
votes
0answers
5 views

Is there a common recommendation on product lifetime declarations in terms of Reliability function?

I mean: if I read on the label that a specific product (car, household appliance, whatever...) is declared to have a lifetime of N years, does it actually mean that the Reliability function R(t), ...
0
votes
0answers
7 views

Cronbach Alpha Assumptions

I'm currently learning ho to use Cronbach Alpha in R. I have a scale with 7 items and 63 respondents. The aim is just to get a practical understanding of what Cronbach Alpha is. There are some nulls ...
0
votes
0answers
17 views

Calculate Cronbach's alpha for three observers

I have a dataset which contains 100 observations, and I asked three expert to describe each observation by one of the letters {A,B,C}. The following three tables shows a sample of the dataset with the ...
1
vote
1answer
34 views

Clear explanation of Cronbach's Alpha

I have difficulty in understanding Cronbach's Alpha's formula. I searched for google and StackExchange able to understand it. Even there are some topics trying to explain it intuitively, I could not ...
0
votes
0answers
7 views

Compute interrater reliability having different subsets of students rated by different subsets of raters

I would like to compute inter-rater reliability for a set of continuous items, having different subsets students assessed by different subsets of raters (please, see the picture). I would like to ...
2
votes
0answers
25 views

McDonalds omega: warnings in R [migrated]

I'm computing omega for several different scales; and get different warning messages for different scales with different omega functions in R. My questions are regarding how to interpret these ...
2
votes
1answer
24 views

Cronbach's alpha vs. Average Item-Sum correlation?

I have heard that Cronbach's alpha is used to measure reliability of a test (to check if a set of items/question are measuring the same construct). How does the Cronbach's alpha compare with the ...
1
vote
1answer
12 views

Randomization scheme for rating images and assessing inter-rater reliability

Apologies for the basic question, but I'm new to the area and I'm working on a research project on rating images, and assessing the reliability of these ratings, with the following structure: There ...
0
votes
0answers
10 views

How to estimate the reliability of an exhaustive list of data?

I have a purported exhaustive list of beneficiaries under a social security scheme. My question is how large must be the sample of this population must be tested to check for fraud etc to derive an ...
0
votes
1answer
16 views

Intrarater reliability test for a single subject?

I have repeatedly measured the volume of the brain of 1 subject in an MRI study. Measurements were acquired by the same rater (myself) on three separate occasions resulting in three measurements of ...
1
vote
0answers
19 views

Effect of reliability on t-test

I have some problem with investigating the effect of reliability on the t-test. Basically it is assumed that lower reliability leads to a broader confidence interval or higher SD, which has a ...
0
votes
1answer
25 views

How can I calculate percent agreement and kappa for each category?

My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges ...
1
vote
0answers
26 views

Goodness of Fit and reliability of coefficients in Logistic Regression

I have logistic regression model with 9 independent variables. Two of them are numerical, the others are binary. Three of my coefficients are significant at 0.001, two of them are significant at 0.05. ...
1
vote
1answer
14 views

What to report for reliability analysis?

After completing data collection on a reliability analysis where multiple examiners sampled each subject three times, which value should I report for descriptive statistics? If intra-rater reliability ...
0
votes
0answers
11 views

Why is my Cronbach alpha result negative

I'm just learning about the Cronbach alpha. I have recently tried to carry out this test. Having 80 completed questionnaires with 15 questions (variables), the questionnaire used the Likert scale 1-5 ...
0
votes
1answer
81 views

Inter-rater reliability measure with multiple categories per item [duplicate]

I have multiple raters providing categorical ratings. What makes this slightly different from other posted questions is that each rater can provide three, unique categorical ratings. Each rater had ...
0
votes
0answers
18 views

Effect of reliability on sample size

I asked a similar question here: Meta-analysis using experimental reliability data However as I did not get any feedback (probably for a good reason, which I would like to understand) I ask the ...
1
vote
0answers
15 views

Combining the standard errors of Cohen's Kappa computed for multiple rater pairs

In a study on intra-rater agreement with 8 raters, I have computed Cohen's Kappa for each of the raters and I estimated the standard error of each of the kappa values using the Jackknife method. To ...
0
votes
1answer
41 views

Normalization of values

How do I normalize a set of data? Let's say we are looking at measurements of diameter of the heart in women and men. Men will have bigger hearts, so error in measuring by the observers will always be ...
0
votes
0answers
6 views

Calculating reliability for scales with cross-loaded items

I am reexamining the second-order factor structure of a well-known multidimensional scale (8 first order factors and 3 second order factors). Using CFA in AMOS I have examined a number of models (I ...
0
votes
0answers
11 views

Negative Cronbach's Alpha

I run the Cronbach's Alpha based on the result of the Exploratory Factor Analysis, and 5 factors found. 1st is positive with Alpha, 2nd is low with Alpha, 3rd, 4th and 5th are having the negative ...
0
votes
0answers
16 views

Lower Cronbach's alpha when adding in reverse coded items

I have been working on some scale development and have administered a series of questions in different phases. The last time I administered the questions I had good structural validity and reliability ...
0
votes
0answers
11 views

Does it make sense to use cronbach alpha to check if questions do NOT pertain to the same construct?

I have three questions in a survey that are supposed to measure different contructs. However, I have my doubts if the respondents will not confuse them. So, I know cronbach alpha is used to check if ...
0
votes
0answers
26 views

Can Cohen's kappa be used for raters with different but related targets?

There are two binary raters: Rater 1 is certified to assess the ability of people to drive a taxi. Rater 2 is certified to assess the ability of people to drive a lorry. Rater 1 and 2 have both ...
0
votes
1answer
26 views

EFA SPSS and Cronbach's Alpha

I have done the EFA in SPSS for my questionnaires and found 4 factors. Now should I use this EFA result to compute the Cronbach's Alpha, and if so then how to do that? Or Cronbach's Alpha is done ...
0
votes
0answers
35 views

Average inter-item correlations as internal consistency indicator

In literature, the average inter-item correlations in the range of 0.15 to 0.50 is considered as an indicator for an acceptable level of consistency. Why do the high average inter-item correlations ...
1
vote
0answers
34 views

How can measure be unreliable but valid?

Some previous research found evidence for the predictive validity of a measure. I've done some work and found the measure to be unreliable. Since reliability provides an upper limit for predictive ...
0
votes
1answer
26 views

Computing Inter-Rater Reliability - Different subjects are rated by different subsets of coders

I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each ...
1
vote
0answers
27 views

Can I draw a single ROC curve for a test method applied onto a group of people with different levels (i.e., light, mild,sever) of sickness

Can I draw a single ROC curve for a test method applied onto a group of people of different levels of sickness severity (i.e., light, mild, severe)? Or I need to draw three different ROC curves, one ...
0
votes
1answer
27 views

Why is Cohen's kappa low despite high observed agreement?

I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can ...
3
votes
1answer
89 views

Mysterious results from likelihood ratio confidence bounds on a Weibull reliability estimate

I'm trying to calculate a confidence interval on an estimate of the cumulative distribution of a two-parameter Weibull distribution. (Actually 1-cdf, the survival probability.) I want to do this using ...
0
votes
0answers
48 views

Cronbach's alpha internal consistency test for interval scale or ratio scale data with different units

I have recently read dos Santos, S. F. & Brandi, H. S. 2015. Model framework to construct a single aggregate sustainability indicator: an application to the biodiesel supply chain. Clean ...
2
votes
2answers
40 views

Is CFA appropriate for test retest reliability?

I know CFA is often used when there is a latent variable that has measurement error that can attenuate correlations. However, does it make sense to make this same adjustment for measurement error when ...
0
votes
0answers
7 views

What is a good book/online source about the subject of reliability?

I am particularly interested about Reliability function $R(t)$ Failure function $h(t)$ Renewal process Maintained systems Nonhomogenius Poisson process and reliability growth Thank you.
0
votes
0answers
24 views

Test-retest reliability

I have come across following table in a publication, where r is correlation coefficient. I could not find detailed procedures for test-retest reliability, is it any different from paired t-test ? ...
2
votes
0answers
91 views

Meta-analysis using experimental reliability data

I have a rather conceptual question. We measured experimentally the reliability of a certain readout. Now what I would like to do is to use this data to give a statement about the results of previous ...
0
votes
1answer
29 views

How to test consistency of responses?

I am analyzing data from several trials where individuals were submitted to different stimuli and a biological response was measured. Some individuals participated in more than one trial where the ...
0
votes
0answers
9 views

Should I rely on the result of Wilcoxon Signed rank or Kruskal Wallis?

I am writing a dissertation on the relationship between age but also investigating other factors such as income, and pro-environmental behaviour. If i test household income against 'Have you taken ...
0
votes
0answers
21 views

Cronbach's Alpha if Item column shows no values

I am doing a report about Factor Analysis, but have no much experience with it. I came up with 3 factors. Factor 1 with 5 items, factor 2 with 2 items and factor 3 with 2 items. I then moved on to a ...
-1
votes
1answer
19 views

how to reliabilty test for this questionnaire

The study on academic stress of students Q1. How frequently are examinations held in your college? a) Every day b) Every week c) Once in a month d) Half -yearly e) ...
0
votes
0answers
12 views

Query on Reliability. The Cronbach's Alpha, Study on Resilience of Vulnerable Children (CYRM-28)

I am doing a research on Risk, Protective factor and Resilience of Vulnerable Children (A study based on Children residing in the Government Children's Homes in Kerala, India). I have used Child Youth ...
0
votes
0answers
15 views

software reliability modeling

Software Reliability Modelling In a given sample data set of discrete software failure data, list of test cases are given and corresponding to that test case, what are the various software bugs ...
0
votes
0answers
19 views

Weibull distribution - reliability prediction

I am using Weibull to predict hard disk failure rates. I used shape 2.0 and scale 1.2 and got fail rates to be [ 1.38888889 2.77777778 4.16666667 5.55555556 6.94444444 8.33333333 9.72222222 ...
0
votes
0answers
13 views

Negative Cronbach's alpha after reversing results of a survey

As part of my master thesis, I am conducting a PCA analysis. The analysis is aiming to find commonalities between 8 survey questions that were answered by 219 respondents. The survey was answered on a ...
1
vote
1answer
61 views

Standard Error of Measurement (SEM) in Inter-rater reliability

I am currently writing my thesis about inter-rater reliability of a diagnostic tool between raters. I want to use the standard error of measurement with the formula: SEM of rater 1 and rater 2 = SD * ...
2
votes
1answer
61 views

Correcting For Reliability/Consistency of a Measure in Meta Analysis (Cronbach's Alpha / Test Re-Test)

Is it good practice to correct for reliability of your measure in a meta analysis? This is what is suggested by Hunter and Schmidt (2004), but it seems weird to me to calculate your average effect for ...
0
votes
0answers
22 views

Categorical Principal Components model summary

I have three questions on CATPCA using SPSS. 1) If I use a varimax rotation, it gives "Model Summary" and "Model Summary Rotation". Should I report values on the Model Summary Rotation since that was ...
1
vote
1answer
40 views

Inter-rater reliability or another method?

I've asked about the mechanics of Inter-rater reliability before, but I guess I should have asked this question instead: If I have a bunch of raters and individuals (who are being judged on ...
1
vote
0answers
27 views

Does measure of reliability apply to regression coefficients?

I have a problem of estimating some kind of a test-retest reliability/intraclass correlation of the regression coefficients. I have only seen test-retest reliability apply to outcome variables, simply ...
0
votes
0answers
11 views

How many raters do I need to determine reliability of agreement on stimulus classification?

I have a stimulus set comprised of 147 videos, each with one of seven labels (hidden from raters). I want to determine how reliable label assignment is for each stimulus -- that is, I want to be able ...