Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

1
vote
0answers
9 views

Measuring equivalence/reliability (ICC and consorts)

For a reliability study I’m administering patient reported outcomes (questionnaires) with 2 modes of administration (on paper and via an electronic device). I’d like to measure equivalence/reliability ...
0
votes
0answers
18 views

Difference between Cronbach's alpha and Pearson's coefficient

I am creating an indicator for social development which includes variables of health, education and economy. Since I have many variables, I decided to remove some of them based on dominion expertise ...
0
votes
0answers
9 views

Inter-rater reliability many coders but only one case

I have a specific performance assessment metric which I aim to get some reliability and validity data for. And I really would appreciate some help for the reliability: Background In my research, ...
1
vote
1answer
46 views

Item response theory for continuous variables, and estimating standard error of measurement

I really like how traditional item response theory (IRT) packages tell you the standard error of measurement conditional on one's ability level, and from that, you can calculate the test information ...
0
votes
0answers
16 views

How to calculate overall reliability for an IRT test with subtests

I have an IRT, 2 parameter model ability test with three subtests - verbal, numerical and abstract reasoning. I have been able to calculate the reliability of each subtest using the methods outlined ...
2
votes
1answer
29 views

Proof of direct proportionality between hazard rate function and probability density function

I'm reading up on reliability and I came across this question: Show that if the hazard function is decreasing, the PDF, $f(t)$, is also a decreasing function and its mode must therefore occur at t =...
0
votes
0answers
19 views

Generating Failure / Suspension data from Weibull Distribution

I've been trying to make some synthetic failure data in MATLAB and I'm coming across an issue when I try to refit the Weibull distribution from my synthetic data. For example, assuming a Weibull ...
0
votes
1answer
10 views

Reliability analysis for different measurement scales?

I'm new here :) I'm conducting a survey questionnaire which gathers feedback on English-as Second Language (ESL). I have two measurement scales in my survey. Questions on how much time they spend ...
0
votes
0answers
29 views

Coefficient omega: appropriate extraction of factors

I use omega_hiearchical to determine the internal consistency of a scale. If I understood it correctly, it tries to extract three lower level factors by default. But you can specify that you want to ...
1
vote
0answers
14 views

Model building, how to include information on data point quality?

I have a dataset with count data predictors and a marker on their reliability or completeness. With 2 options: 0 - Data is not confirmed (could be higher); 1 - data is confirmed to be complete. The ...
0
votes
0answers
9 views

Best way to measure reliability of auto-correlated data

I'm trying to compare different ambulatory devices for measuring heart rate against a gold standard in-lab assessment. What is the best way to determine whether a device is worth using. I can use ...
0
votes
0answers
15 views

How much time is needed between measures for test-retest reliability?

I am planning a meta-analysis on reliability of various loneliness scales. We want to determine beforehand what time frame between measures we will consider. For instance, if the time between 2 ...
1
vote
1answer
74 views

Connection between Hazard Ratios, Survival, and Probability

According to this wiki link, the hazard ratio relates to survival according to the following equation $$ S_1(t)=S_0(t)^r \quad (1) $$ where $S$ is the survival and $r$ is the hazard ratio. So from ...
0
votes
0answers
17 views

Evidence fusion-Dempster Shafer

I have developed four models that estimate the same state-traffic volume in 100 streets during a specific time period. I have actual volume for 25 streets to estimate compute the reliability of each ...
0
votes
0answers
31 views

Can VAR models capture all frequency interactions reliably and to the same extent?

I would like to assess how a set of ~50-100 time-series variables causally interact with each other. To do this, I am fitting a VAR model and using cross-validation to estimate model-order. After the ...
0
votes
1answer
27 views

In scale adaptation which should be done first? CFA or Reliability Analysis

I am adapting questionnaire cross-cultural. Questinonnaire has 11 item and two factor. I want to confirm the structure 1 item's corrected item total correlation is <.20. Should this item be ...
0
votes
0answers
10 views

Measurement theory and sample size calculation for multivariate testing

Let $\mathbf{Y}$ be a vector of independent, normally-distributed random variables. Let $S_1$, $S_2$ and $S_3$ be three non-overlapping samples of sizes $N_1$, $N_2$ and $N_3$, respectively. Let $M_A$ ...
1
vote
0answers
12 views

Is it correct to a total Cronbach´s alpha analysis to a tridimensional scale?

There´s a statement that says that Cronbach´s alpha assumes unidimensionality. In that sense, is it correct to calculate a global alpha? I mean, by including all 3 dimensions, and not by separating ...
1
vote
0answers
50 views

Can we statistically predict the next Lion Air incident? [closed]

Plugging the Lion Air incidents on Aviation Safety into a Crow/AMSAA tool gives MTBF of about 15 months at the moment. Is this reasonable for their fleet size of 115? Link to the: incident data ...
1
vote
0answers
16 views

Reliability of Composite Variable Made of 4 Measures?

I am conducting some analyses with a composite variable constructed by averaging 4 separate measures. I would like to know the reliability of the composite variable. Is Cronbach's alpha appropriate in ...
0
votes
1answer
92 views

Hazard function for a growing population

I want to model the expected failure rate in a growing population. Each individual has a high infant mortality (e.g. Weibull(shape=0.5, scale=100)). So the ...
1
vote
2answers
57 views

Book reference request for statistical reliability theory

I'm about to learn statistical reliability theory for the first time. I'm told that Barlow and Proschan's book entitled "Statistical theory of reliability and life testing" is a classic on this topic. ...
1
vote
0answers
92 views

Best easy and straightforward way of computing McDonald's omega in a unidimensional test with 'psych' in R

I'm preparing a masterclass in Psychometrics for a course, and I would like to introduce McDonald's omega coefficient as the proper index for assessing the reliability of a scale. For what I can see, ...
1
vote
0answers
344 views

McDonald's Omega: Assumptions, Coefficients and Interpretation

After browsing cross validated and several other sources on the web, I still cannot get a grip on McDonald's Omega as a measure of internal consistency. I have a hunch that many fellow social ...
1
vote
0answers
32 views

Index of within-participant consistency in two-alternative forced-choice (2AFC) task?

My study consists of 2AFC trials: Participants choose which one of a pair of stimuli has a certain quality. There are 9 pairs and 6 stimuli (2 categories of 3 stimuli each) in total. The stimuli ...
0
votes
1answer
29 views

How to calculate Mean Sample-Based Reliability Estimates Used for Analyses

I am conducting a meta analysis of Pearson's r , of correlations presented between test scores and job performance ratings. I need to present an uncorrected effect size and an effect size corrected ...
0
votes
1answer
121 views

Cronbach alpha with knowledge items?

I have a very specific question about the application of Cronbach's alpha to data with trichotomous responses (true-false-don't know). I have an acquaintance I'm helping replicate a study using a 45 ...
1
vote
1answer
34 views

Intraclass Correlations for Intraobserver-Reliability in Hierarchical Data

We have a dataset where multiple measurements of the same variable are taken from individuals and we want to measure the intraobserver reliability. However, the individuals each come from different ...
0
votes
0answers
112 views

Test-Retest reliability for mixed models

We have an experimental setup as follows: A set of subjects each repeat a single task multiple times in a single session, yielding a set of measurements. The same subjects then repeat the same task ...
0
votes
0answers
16 views

Cronbach's Alpha to Rate Videos

I would like to see if we can have reliable metrics that can be used to rate videos (V1:V15). I have 20 people for each video (id1:id20). And 2 metrics: ...
0
votes
0answers
48 views

Cronbach's alpha and yes-no-don't know scales

I am helping a friend analyze some questionnaire data, and I am currently a bit confused about how I should go about doing reliability analysis on her items. She is using a 45 item scale, assessing ...
0
votes
0answers
30 views

Reliability calculation using R

I have this handout about reliability calculation, but I just can't figure out how I could do this using R. Here is the simulated data. It is a questionaire that has ten quaestions and was answered ...
0
votes
0answers
15 views

Seeking a measure of multiple rater consistency for cateorical ratings with sparse data

I have multiple raters and sparse data. The raters can decide how many attributes apply to a person. I have many empty cells due to many attributes and few attributes actually chosen. What tools ...
5
votes
2answers
109 views

How to index rater agreement: multiple raters identify strenghts/weaknesses from 30 traits?

Six raters independently rated 25 people by identifying strengths and weaknesses: choosing among 35 traits (15 positive traits and 20 negative traits). The rater instructions said to identify a ...
1
vote
0answers
103 views

What is the point of reporting split-half reliability together with Cronbach's alpha?

In several journal articles I read, autors report split-half reliability (Guttman formula obtained if questionnaire is splited into two parts: odd items and even items, usually) together with Cronbach'...
0
votes
0answers
38 views

Derivation of equations for confidence interval of mean of a Poisson distributed variable

I'm having trouble understanding why the first equation found in the following Wikipedia link is valid: https://en.m.wikipedia.org/wiki/Poisson_distribution#Confidence_interval Can anyone here derive/...
1
vote
1answer
257 views

Using Cohen's kappa for multiple item types (e.g. binary and non-binary)

I am looking to calculate Cohen's kappa for multiple items in a scale (15 items, with two raters). The problem is that for 12 of the items the responses available to the raters were binary (yes, no). ...
0
votes
0answers
56 views

Can we use cronbach's alpha reliability test to check a variable scale with 3 question, same 7 point scale but different values?

I am a research student and doing my first paper. I am confused about checking the reliability of my adopted scale. I am using TAM as a theoretical model for my research and for it I adopted the ...
0
votes
0answers
37 views

Device that improves with time

I am unable to find an example of a device that improves with time, i.e., every time you use the device, the probability of it failing is less than before. I understand that it is an example of ...
0
votes
0answers
19 views

Test-retest Reliability Before and After

Is there some way I can assess test-retest reliability for a questionnaire based on pre and post-treatment results for patients? They are both using the same questionnaire, but the patients are in the ...
0
votes
0answers
105 views

What qualifies as a good split-half relibability?

We were doing an sentiment annotation task for natural language processing and achieved a split-half reliablity of about r=.9. I am rather confident that this is a good score, but I would like to have ...
1
vote
0answers
35 views

Analyzing satisfaction levels of PhD scholars

I am analyzing satisfaction levels of PhD scholars in a particular region. We got 7 independent variables from literature review and constructed 45 questionnaires accordingly: for example, 5 ...
1
vote
0answers
24 views

What would be an appropriate intra-rater reliability method for just 3 items test-retest

I had 20 raters evaluating 35 items with continuous interval scores. 3 items (randomly chosen out of the 35 for each rater) were tested at the beginning and then again at the end of the session. I ...
1
vote
0answers
95 views

how to calculate inter-rater agreement of nominal data with multiple categories in r

coder1 results coder2 results Hi, what method should I use to calculate inter-coder reliability between these two data in r? The row stands for the cases, and the a1~ a23/A1~A23 stands for the ...
1
vote
1answer
115 views

error variance (delta) of a factor - lavaan

I'm building a measurement model with three factors and I don't know how to set the error variance ((1 - reliability) * item's variance) of one of the factors in lavaan. Thanks for your help.
2
votes
1answer
40 views

Does changing the point range of items of a construct measured through likert scale affect in any way validity and reliability of the construct?

I am using a pre-existing universal tested scale to measure a specific construct. The construct is measured through 5 items on a 5 points likert scale. However, I would probably need to change the ...
4
votes
2answers
234 views

The dirty coins and the three judges

All things being perfect, if we want to know if a set of coinage (all coins with the same bias) is biased, and to what degree, we would just use the outcomes and confidence intervals of the binomial ...
0
votes
1answer
18 views

determining validity of the regression

I have performed curvilinear regression analyses on my data and the curvilinear fit is better than the linear (as determined by the F-stat of anova). I am now searching for a way to measure the ...
0
votes
1answer
77 views

Minimal detectable change based on >2 measurements

I need to calculate the minimal detectable change (MDC) for my dataset and found the following formula in several publications: $MDC = 1.96 * SD * \sqrt(1-ICC) * \sqrt2$ They argue $\sqrt2$ is used, ...
2
votes
1answer
100 views

Which ICC model to use for computer vs. human agreement

I have an agreement study with the following basic construction: ~ 400 items are rated by (1) a computer algorithm, and (2) a human rater. The ratings are on a continuous scale. I would like to ...