Questions tagged [reliability]
A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater
528
questions
0
votes
0
answers
24
views
Best way to perform a reliability growth analysis and answering some questions like "the reliability is getting better?"
everyone!
I'm a reliability engineer and a project was given to me two weeks ago. I need to answer some questions about the reliability of a specific item in a train. There are 189 items, one for each ...
0
votes
0
answers
10
views
What type of inter-rater reliability analysis should I do?
I have 16 pairs of raters. Each pair rated 5 items from one participant. There were a total of 16 participants. What type of inter-rater test should i do?
- 11
1
vote
1
answer
24
views
Combining two success runs in parallel
The goal is to calculate the reliability of a process. Here reliability is defined as follows:
Definitions and tests I used
Let $X$ be a random variable that is equal to $1$ when no defect is present ...
- 427
1
vote
1
answer
26
views
Compare reliability / lifetimes/ failures of two different groups
I have two different groups of devices with different failure rates:
Group 1: 72 devices, 57 % did not fail during observation.
Group 2: 11 devices, no device failed during observation.
How could I ...
- 3,123
0
votes
0
answers
8
views
Inter-rater reliability against multiple raters
I am conducting inter-rater reliability by doing a clinical test of 25 participants. Can I do the reliability against multiple raters? I mean that I will do the test for all the participants and then ...
1
vote
0
answers
23
views
Optimal Intraclass correlation coefficient computation method
I am hoping to compute intraclass correlations to evaluate interrater reliability (IRR) for interviews conduct with participants in a clinical psychology study. Specifically, we want to determine IRR ...
1
vote
1
answer
31
views
How to measure reproducibility for repeated measures
I have run an experiment where in a series of trials, each of 10 animals was presented with each of 6 stimuli, and each stimulus was offered once to the left and once to the right nostril. Overall, ...
- 11
0
votes
0
answers
19
views
What does MSE mean in the calculation of separation reliability for rasch model?
In the Rasch model, the reliability of separation can be calculated as 1 - (MSE / SD^2), where SD represents the standard deviation of person location measures on a logit scale, and MSE refers to the ...
- 33
0
votes
0
answers
34
views
Bayesian model for the reliability of a manufacturing process with noninformative prior
We have a manufacturing process $M$ with an unknown reliability $R \in (0,1)$, and, at the end of it, an automatic tool that sorts the good and bad products with a known efficiency of $E \in (0,1)$. ...
- 427
0
votes
0
answers
19
views
Can you use Krippendorff's Alpha to assess reliability of an entire measure using one case study?
I am interested in using Krippendorff's Alpha for a dataset containing a large number of raters to assess the reliability of using an assessment tool for case studies, which contains 8 items with ...
- 1
2
votes
0
answers
36
views
Some Question about the Intraclass Correlation in interrater reliability
I am currently working on intraclass correlation in the context of inter-rater reliability. There are different forms of ICC (Shrout and Fleiss, 1979; McGraw and Wong, 1999) and according to the ...
- 93
1
vote
1
answer
30
views
Hypergeometric CDF in R
I'm working on a reliability problem, and I've run into a snag. I have a population of 300 widgets. From that population, I took a random sample of 10 and 1 turned out to be bad. What I'd like to know ...
- 115
1
vote
1
answer
17
views
Approach of a Golden Standard Measurement with questionnaire data
I am a bit lost on the correct approach to this Statistical problem. Let me explain the context of this problem: I have a cohort population with longitudinal measurements during a period of several ...
- 11
1
vote
0
answers
252
views
Creating and interpreting calibration plots for several models with a binary outcome
I have made several models (RF, XGB and GLM) to predict a binary outcome and they all achieved an AUC of approximately 0.8 and Brier scores 0.1-0.15.
Test set is fairly small (n= 350), cases with ...
- 11
0
votes
0
answers
14
views
Can I delete a dimension of a scale if this dimension reach a very low internal consistency when doing psychological network analysis
I am trying to do a psychological network analysis, and I aim to do the analysis on dimension-level (summing all the items belonging to this dimension). However, I've found that one dimension of the ...
- 15
1
vote
1
answer
28
views
Right-censoring Survivability with Custom Likelihood Function
I am having bit of difficultly understanding the process of using censored data. I know there are plenty of R packages that can do this for the usual two-parameter Weibull likelihood formula, but I ...
- 199
0
votes
0
answers
7
views
Comparing scale worded slightly differently across 3 datasets
I am working with 3 cohorts, 2 of which received a 5-item measure (say measure A) and one that received a modified version of the same measure (measure B). Essentially, measure B is comprised of same ...
0
votes
0
answers
25
views
ROC analysis with a fuzzy reference standard with estimates of its reliability
I‘m conducting ROC analyses in order to assess the diagnostic accuracy (AUC, sensitivity and specificity for certain cut-offs) of multiple index tests (ordinal scaled measures). The goal is to compare ...
1
vote
1
answer
52
views
Chance Agreement for Fleiss Kappa
In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as
where $p_j$ is defined as the proportion of ratings which were to ...
- 31
2
votes
1
answer
136
views
Reliability of 3-way interaction term between correlated variables
If we have two variables, $X$ and $M$, which follow a multivariate normal distribution, have a mean of 0, are correlated ($r_{XM}$), and have known reliabilities ($\rho_X$ and $\rho_M$), then the ...
- 1,069
0
votes
0
answers
18
views
Measuring inter-rater agreement for unclear number of events
I have 2 raters who watch videos of test sessions and annotate emotions. There is a defined number of categories to choose from. But there is no defined number of situations they have to annotate. So ...
- 1
1
vote
0
answers
14
views
Equipment reliability in medical device manufacturing setting
I am working in a medical device company and there is something that I would like to test out on the manufacturing line. I am not a statistics expert so this is something that is out of my skillset.
...
- 11
1
vote
1
answer
244
views
Reliability of yes or no questionnaire
I have a questionnaire of 15 questions with "yes" or "no" responses distributed to 20 participants. It will be used as an index measure of a variable. How do I measure the ...
- 13
0
votes
0
answers
132
views
Is the coefficient of determination (r-squared) a measure of reliability or validity?
I am working on my thesis where I want to assess the reliability and validity of different regression models. I only have one data set, which means that I cannot train on one data set and test on ...
- 13
0
votes
0
answers
82
views
How to do sample size calculation for Krippendorff's alpha (paired nominal/categorized data with 10 raters/observers)?
I'm planning an prospective observational trial studying the agreement of 10 doctors (from the same field) in diagnosing certain findings on ultrasound (nominal yes/no outcomes). There will inevitably ...
- 11
3
votes
1
answer
657
views
What is cor.smooth(R) : Matrix was not positive definite warning Cronbach alpha in Psych?
I'm getting the warning In cor.smooth(R) : Matrix was not positive definite, smoothing was done, but what is it in this case? Can I get away with that?
code:
<...
- 641
0
votes
0
answers
22
views
Is there an accepted equivalent of Cronbach's alpha when there are more items than cases
I have a test with 160 items (with binary responses), and have 135 cases (and no way of collecting more, this a dataset where collection has already ended).
Reviewers have asked for a reliability ...
- 161
1
vote
1
answer
22
views
Data imputation for incorrect answers
Problem: I have an MCQ language proficiency test completed by 147 participants. 6 questions were never answered correctly by all participants, which left 30% of the test data as NA. I want to assess ...
- 13
0
votes
0
answers
14
views
How to use standard error of measurement when comparing two individuals' scores and evaluating interventions
Comparing Individuals
Scenario: Let's say two individuals, A and B, take some assessment. A scores 60, and B 65. This assessment has a standard error of measurement of 10.
Question: Can we say with ...
0
votes
1
answer
41
views
Should we test only 20% of the sample for intraclass correlation coefficient?
I'm currently calculating an intraclass correlation coefficient for my data. As far as I'm concerned, the best choice (following this approach and the approach kindly recommended on the previous post ...
- 641
2
votes
1
answer
127
views
inter-rater reliability for numeric variables
I've seen some questions on inter-rater reliability here for categorical variables. I'm wondering what would be an adequate approach to test inter-rater reliability between two raters for a numeric ...
- 641
1
vote
1
answer
32
views
Inter-rater reliability (complex survey )
Help calculating Inter-rater reliability in a complex online survey (free text).
The questions are free text. We have now coded the potential answers. Examples:
1.) Hobbies (Time in nature, mind-body ...
- 11
0
votes
0
answers
8
views
how can I assess internal consistency for an algorithmic classification?
Is there a way to assess internal consistency for an algorithmic classification based on a rating scale? Or other appropriate reliability metrics?
For instance, let's say that based on a 6-item, ...
- 190
2
votes
1
answer
408
views
Mean Residual Life of Weibull Distribution closed form
Weibull distribution function is parameterized as $ F(x) = 1 - e^{(-(x/\sigma)^a)}$ and survival function is $S(x)= e^{(-(x/\sigma)^a)}$ where $a$ is shape parameter and $\sigma$ is the scale ...
- 7,993
2
votes
1
answer
150
views
Can I check the reliability of my test after collecting my data?
I have performed a study where I tested 2 groups, before and after two different types of manipulations. The main outcome of my study were the group differences at time point 2.
Now someone asks what ...
- 289
0
votes
0
answers
70
views
Group sequential design, repeatability/ reliability/ intra-class correlation coefficient (ICC) and sample size calculation
I want to test a survey taken by two nurses on the number of patients and determine the intra-class correlation coefficient/reliability. I want to determine the sample size for the analysis which will ...
- 1
0
votes
0
answers
11
views
How to determine the variation of influences?
I need to investigate a chain of influences which can be described as following: First, a deliverer sends barrels of oil. These will be connected to a certain system and purified. Next, the oil will ...
- 3,123
1
vote
1
answer
276
views
Lifetime vs mean time between failure (MTBF)
I'm wondering about the difference between MTBF and lifetime (I've to admit I'm not sure whether the last link is too specific..).
source
source
To make it precise: Let's say there is a single ...
- 3,123
2
votes
1
answer
72
views
How to interpret the Rasch reliability?
I am tasked to model a set of questionnaires in order to investigate about the food insecurity in the region where my project takes place. Using the Rasch model, I was able to conduct the protocol ...
- 243
2
votes
1
answer
40
views
Do you have to assess reliability for non-composite scores?
I feel like this is probably a really stupid question but I'm having issues finding the answer. When I think of reliability, normally I think of several items on a test that are compiled into some ...
- 6,021
4
votes
2
answers
163
views
Does using a Likert Scale require me to always measure the internal consistency?
I am doing a Masters project (in environmental sociology) which includes a series of 15 questions that use a 6-point Likert scale to respond to various statements. In my pilot survey, my supervisor ...
- 101
1
vote
0
answers
226
views
What are omega 1 and omega 2 in the reliability output from R?
I read this very useful article on omega reliability coefficients and have since been toying around with it in R (using the reliability function from the ...
- 6,021
1
vote
1
answer
178
views
How to calculate categorical omega reliability coefficient when some items only have one value?
From what I understand, running omega reliability in R is quite straightforward, even for binary responses. However, I've encountered a situation when running this command:
...
- 6,021
3
votes
0
answers
172
views
What are the rules of thumb for interpreting correlation coefficients corrected for unreliability?
I performed a series of meta-analyses in which we corrected correlation coefficients for measurement unreliability. A reviewer is asking us to define the magnitude of the meta-analytically-produced ...
- 363
1
vote
0
answers
27
views
Reliability of opensource ML packages [closed]
While working on a machine learning project, in addition to the widely known opensource ML packages like tidyverse (R), pandas/scikitlearn (python), we end up using lot of other little known packages, ...
- 91
2
votes
2
answers
291
views
Cronbach’s alpha not equal to ICC 3,k
I am calculating ICC over 16 items rated by 41 raters, using the ICC function from the psych package. The output is:
...
- 21
1
vote
1
answer
46
views
Inter-rater reliability
In my research, I use a questionnaire so that students mention some classmates according to a group of characteristics. Each item is a different characteristic, and for each one they must name the ...
1
vote
0
answers
161
views
Computing IRT score reliability across a range of ability in mirt
I am using mirt and would like to get reliability estimates across a range of ability for a 1-factor scale. Essentially, I would like to extract the data used to ...
- 11
2
votes
0
answers
50
views
Composite reliability for a simple parallel model
I want to calculate the composite reliability for a simple parallel model as follow.
And the below is what the textbook says (you can skip this part if you want)
When X is a composite score, using A ...
- 143
0
votes
0
answers
27
views
How do I calculate different reliabilty levels of theta when standard error of theta is known? (IRT-model) [duplicate]
I've recently seen a plot where the level of reliability is given for different SE values of theta estimates:
I have also plotted the standard errors of theta as a function of theta but now I am ...
- 31