Skip to main content

Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

Filter by
Sorted by
Tagged with
1 vote
0 answers
22 views

Interpretation of ICC and Pearson's r correlation

I am calculating intraclass correlation coefficient (ICC) values for response values at two different timepoints. I am also correlating the response values at the different timepoints using Pearson's $...
Lee's user avatar
  • 11
2 votes
2 answers
145 views

Question on the effect of test reliability on weighting of a test battery

Originally, a test battery had 4 components: two 100 item multiple-choice tests, one oral test, and one essay test. Each component measured different topics. Each of the 4 components was weighted 25%. ...
Joel W.'s user avatar
  • 3,375
1 vote
0 answers
63 views

Calculate inter-rater noise using Kahnemans (2021) approach

I need help calculating signal and noise based on the method described by Kahneman et al. (2021) in their book "Noise." They provide a technique for quantifying noise between raters ...
Magnus Nordmo's user avatar
1 vote
1 answer
32 views

Expected test reliability for short, multiple-choice tests?

How might I calculate the expected test reliability (internal, e.g., KR-20) for tests of various lengths? What assumptions would underlie such calculations? Are there existing tables or calculators ...
Joel W.'s user avatar
  • 3,375
1 vote
0 answers
20 views

How to calculate an ICC for test-retest, with more than one rater

In my data, i have two raters that rated multiple scores to do a test-retest analysis. The method fulfill the requirements for test-retest (short period between both visits) I would like to calculate ...
BPeif's user avatar
  • 143
3 votes
1 answer
21 views

Incorporate retest reliability into model (in R)

I'm conducting a study to evaluate the impact of an intervention on a test outcome. Each participant takes the test before and after the intervention. Also there's a control group involved. Given that ...
Jailbone's user avatar
0 votes
0 answers
12 views

Is it possible to calculate inter-rater reliability for one item rated by multiple raters with weights?

I have a survey with a number of statements that had participants categorizing the statements into one of the 4 options they were provided. The participants were then asked to rate the confidence of ...
Jay Jakka's user avatar
2 votes
1 answer
35 views

Why is reliability a ratio of true score variance to total score variance, instead of the ratio of true score to total score?

Reliability is given by: $r_{xx} = \frac{Var(T)}{Var(X)}$ where $T$ is the true score and $X$ is the total score. Why isn't reliability simply: $r_{xx} = \frac{T}{X}$ where $T$ would be a sum of all ...
J. Doe's user avatar
  • 357
0 votes
0 answers
23 views

What is known about the (in)efficiency of true score variance estimation?

In classical test theory, an observed score $X$ is defined as $$ X = T + E $$ where $T$ represents the individual true score and $E$ represents the individual error score. In this context $T$ may be ...
LJ Beinhauer's user avatar
0 votes
0 answers
42 views

Repeatability coefficient and Standard error of Measurement

The repeatability coefficient (CR) (Bland Altman) is used to report repeatability understood as the index that compares measurements with the same subject, the same observer and in a short space of ...
mdscience's user avatar
0 votes
0 answers
20 views

How to format data for one-way intraclass correlation coefficient?

I'm testing inter-rater reliability of an instrument on a sample of subjects from several sites. Each subject is assessed by two raters, different between each site. According to this paper, I should ...
Charlie's user avatar
  • 142
2 votes
0 answers
22 views

What test of reliability for repeated measurements of the same scale?

I need to ask you for help on what method to use to calculate the reliability and validity of a scale that I measured repeatedly on the same subjects during an experiment. Let me explain: normally ...
Ric87's user avatar
  • 71
0 votes
0 answers
10 views

Validity, reliability and repeatability

I want to validate an anemometer in a wind tunnel VS gold standard. Thinking in validity, reliability and repeatability. The dependent variable is wind speed. I can test 3 different units of the same ...
mdscience's user avatar
0 votes
0 answers
20 views

How to calculate reliability of difference scores?

I am trying to calculate the reliability of a difference score. Specifically, the data have, for each participant, scores for 10 items in Condition X (1s and 0s), as well as 10 different items in ...
Altair555's user avatar
0 votes
0 answers
48 views

Test-retest reliability in R and changes in score of the individuals

64 cows, subjectivelly tested for chute score response in 2021, by one observer; from score 1 (calm response) to score 4 (nervous response). The same test in 2022, same observer, same animals. Test-...
Madalina Mincu's user avatar
3 votes
3 answers
335 views

Cronbach alpha - descriptive statistics

I researched thinking styles based on Sternberg's Investment Theory. I used a TSI questionnaire that measures preferences in thinking styles. The problem is that I got low scores of Cronbach's alpha ...
goldy's user avatar
  • 31
2 votes
0 answers
34 views

Is there a statistic that expresses whether two measures tell us the same information?

After trying to understand different implementations of intraclass correlation (ICC), inter-rater reliability, and bland-altman statistics, I'm not quite sure what is the right choice here, and would ...
stck8888's user avatar
0 votes
0 answers
56 views

Good reliability analysis with reverse coded items: further checks needed?

After conducting an exploratory factor analysis on a set of data, I have obtained 3 factors, whose items (can) make sense from a theoretical perspective — even though some negatively worded items ...
Tom's user avatar
  • 7
2 votes
1 answer
92 views

Inter-rater reliability for mixed data types?

I am very new to statistics, and I need help determining the the best inter-reliability test for a questionnaire. My questionnaire will be assessed by three raters. I'm just providing the sample of my ...
Ashti's user avatar
  • 63
0 votes
1 answer
45 views

How do we decide the sample size we need for ensuring reliability for a test?

There is a test consisting of 6 questions and additional sub-Qs. The test was translated into another language and the questions were adapted, but the reliability of the test was never assessed. The ...
jordanLeah's user avatar
1 vote
0 answers
67 views

Measuring the reliability of a survey data

I have a survey data and I applied KR20 on it. The KR20 score is 0.63 which means this survey result is not consistent and reliable, at least not in a reliable range with the definition of a reliable ...
Moh-Spark's user avatar
1 vote
1 answer
115 views

How to compare Gwet's ac1 coefficient between group of raters?

So I have 4 raters who scored the subjects in the study with a nominal score like: "Present", "maybe", and "absent". I made pairs of the raters and evaluated their inter-...
Kierrajames's user avatar
2 votes
2 answers
270 views

Negative Cronbach's alpha – recode data, or use another method?

Below is a made up scenario to exemplify my issue. I have two questionnaires asking individuals how much headache they have compared with 7 days ago. The scale is from much less through little less, ...
Wandering_geek's user avatar
2 votes
1 answer
37 views

Is there a proof of the equivalence between Cronbach's alpha and reliability calculation from a CFA?

Given a uni-dimensional CFA, reliability can be calculated as: $$ \omega = {\left(\sum_{i=1}^p \lambda_i\right)^2\over \left(\sum_{i=1}^p \lambda_i\right)^2 + \sum_{i=1}^p\sigma^2_i} $$ where $\...
Jayden Nord's user avatar
0 votes
0 answers
26 views

Comparing proxy metrics to human evaluations

I have two proxy metrics, and I'd like to see which of them correlates more strongly with human ratings. I have ~30 questions, and for each question 3 humans independently give a score on a 1-10 scale....
augray's user avatar
  • 101
2 votes
3 answers
431 views

How to calculate Cronbach's Alpha for a total scale that consists of four subscales?

I have a scale that has 4 subscales. Each subscale has 5 items. For calculating Cronbach's alpha for the total scale, should I use subscale values (in this case it would be 4) or should I use ...
karen's user avatar
  • 21
0 votes
0 answers
13 views

Bootstrap for Fleiss' K reliability

I have a dataset with 5 raters and 129 nominal observations. I want to calculate Fleiss'K for inter-rater reliability and I need to use bootstrap to calculate the Confidence Intervals. I am using R ...
FrAiello's user avatar
5 votes
1 answer
239 views

Time-Dependent Probabilities and Survival Analysis

I am working on a data-driven collision risk model for cars. Using car trajectory data, the model computes the probability of collision at each data timestamp with 2 sub-models: An overlap model: By ...
bfgt's user avatar
  • 317
0 votes
0 answers
37 views

Best way to perform a reliability growth analysis and answering some questions like "the reliability is getting better?"

everyone! I'm a reliability engineer and a project was given to me two weeks ago. I need to answer some questions about the reliability of a specific item in a train. There are 189 items, one for each ...
Caio Rodrigues's user avatar
0 votes
0 answers
13 views

What type of inter-rater reliability analysis should I do?

I have 16 pairs of raters. Each pair rated 5 items from one participant. There were a total of 16 participants. What type of inter-rater test should i do?
em123's user avatar
  • 11
1 vote
1 answer
41 views

Combining two success runs in parallel

The goal is to calculate the reliability of a process. Here reliability is defined as follows: Definitions and tests I used Let $X$ be a random variable that is equal to $1$ when no defect is present ...
lulufofo's user avatar
  • 472
6 votes
1 answer
81 views

Compare reliability / lifetimes/ failures of two different groups

I have two different groups of devices with different failure rates: Group 1: 72 devices, 57 % did not fail during observation. Group 2: 11 devices, no device failed during observation. How could I ...
Ben's user avatar
  • 3,503
1 vote
0 answers
31 views

Optimal Intraclass correlation coefficient computation method

I am hoping to compute intraclass correlations to evaluate interrater reliability (IRR) for interviews conduct with participants in a clinical psychology study. Specifically, we want to determine IRR ...
Wyoming Psychology's user avatar
1 vote
1 answer
45 views

How to measure reproducibility for repeated measures

I have run an experiment where in a series of trials, each of 10 animals was presented with each of 6 stimuli, and each stimulus was offered once to the left and once to the right nostril. Overall, ...
Aga's user avatar
  • 11
0 votes
0 answers
36 views

Bayesian model for the reliability of a manufacturing process with noninformative prior

We have a manufacturing process $M$ with an unknown reliability $R \in (0,1)$, and, at the end of it, an automatic tool that sorts the good and bad products with a known efficiency of $E \in (0,1)$. ...
lulufofo's user avatar
  • 472
2 votes
0 answers
70 views

Some Question about the Intraclass Correlation in interrater reliability

I am currently working on intraclass correlation in the context of inter-rater reliability. There are different forms of ICC (Shrout and Fleiss, 1979; McGraw and Wong, 1999) and according to the ...
バシル's user avatar
  • 123
1 vote
1 answer
32 views

Hypergeometric CDF in R

I'm working on a reliability problem, and I've run into a snag. I have a population of 300 widgets. From that population, I took a random sample of 10 and 1 turned out to be bad. What I'd like to know ...
Karl Wolfschtagg's user avatar
1 vote
1 answer
19 views

Approach of a Golden Standard Measurement with questionnaire data

I am a bit lost on the correct approach to this Statistical problem. Let me explain the context of this problem: I have a cohort population with longitudinal measurements during a period of several ...
mdelas's user avatar
  • 11
1 vote
0 answers
673 views

Creating and interpreting calibration plots for several models with a binary outcome

I have made several models (RF, XGB and GLM) to predict a binary outcome and they all achieved an AUC of approximately 0.8 and Brier scores 0.1-0.15. Test set is fairly small (n= 350), cases with ...
mmo's user avatar
  • 11
1 vote
1 answer
40 views

Right-censoring Survivability with Custom Likelihood Function

I am having bit of difficultly understanding the process of using censored data. I know there are plenty of R packages that can do this for the usual two-parameter Weibull likelihood formula, but I ...
coolhand's user avatar
  • 199
1 vote
1 answer
104 views

Chance Agreement for Fleiss Kappa

In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as where $p_j$ is defined as the proportion of ratings which were to ...
crabnebul's user avatar
2 votes
1 answer
196 views

Reliability of 3-way interaction term between correlated variables

If we have two variables, $X$ and $M$, which follow a multivariate normal distribution, have a mean of 0, are correlated ($r_{XM}$), and have known reliabilities ($\rho_X$ and $\rho_M$), then the ...
David B's user avatar
  • 1,612
1 vote
0 answers
14 views

Equipment reliability in medical device manufacturing setting

I am working in a medical device company and there is something that I would like to test out on the manufacturing line. I am not a statistics expert so this is something that is out of my skillset. ...
Eugene Ho's user avatar
1 vote
1 answer
959 views

Reliability of yes or no questionnaire

I have a questionnaire of 15 questions with "yes" or "no" responses distributed to 20 participants. It will be used as an index measure of a variable. How do I measure the ...
Jof frimpong's user avatar
0 votes
0 answers
325 views

Is the coefficient of determination (r-squared) a measure of reliability or validity?

I am working on my thesis where I want to assess the reliability and validity of different regression models. I only have one data set, which means that I cannot train on one data set and test on ...
Evi S's user avatar
  • 13
3 votes
1 answer
1k views

What is cor.smooth(R) : Matrix was not positive definite warning Cronbach alpha in Psych?

I'm getting the warning In cor.smooth(R) : Matrix was not positive definite, smoothing was done, but what is it in this case? Can I get away with that? code: <...
Larissa Cury's user avatar
1 vote
1 answer
26 views

Data imputation for incorrect answers

Problem: I have an MCQ language proficiency test completed by 147 participants. 6 questions were never answered correctly by all participants, which left 30% of the test data as NA. I want to assess ...
Alaa's user avatar
  • 23
1 vote
1 answer
120 views

Should we test only 20% of the sample for intraclass correlation coefficient?

I'm currently calculating an intraclass correlation coefficient for my data. As far as I'm concerned, the best choice (following this approach and the approach kindly recommended on the previous post ...
Larissa Cury's user avatar
2 votes
1 answer
417 views

inter-rater reliability for numeric variables

I've seen some questions on inter-rater reliability here for categorical variables. I'm wondering what would be an adequate approach to test inter-rater reliability between two raters for a numeric ...
Larissa Cury's user avatar
1 vote
1 answer
41 views

Inter-rater reliability (complex survey )

Help calculating Inter-rater reliability in a complex online survey (free text). The questions are free text. We have now coded the potential answers. Examples: 1.) Hobbies (Time in nature, mind-body ...
Vit Frac's user avatar

1
2 3 4 5
12