Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

0
votes
0answers
11 views

Reliability resulting from Classification + Discriminant Analysis

I have the task to classify 90 small lakes into one of the classes "very salty", "moderately salty", "not salty". I use two different methods to do that. Method 1 is based on Electric Conductivity ...
4
votes
0answers
144 views

Intraclass correlation coefficient (ICC) for two raters using a mixed effects model for a design with repeated measurements

I want to calculate the intraclass correlation coefficient (ICC) for an inter-rater assessment with two raters. The design is as follows: Each of the $n$ subjects were assessed by the same 2 raters on ...
0
votes
0answers
12 views

Which ICC (ICC(2,1) or ICC(2,k)) should I report?

I have some hundred subjects indicating on a 7point scale for 10 situations how likely they are to be influenced in diverse situations. I will build an index of the responses for these behaviors and ...
0
votes
0answers
14 views

Tau, congeneric and parallel indicator

I'm trying to calculate with r a tau, congeneric and parallel model starting from a matrix correlation but I don't seem to manage to get it right. I'v particular trouble with the tau equivalent model. ...
0
votes
0answers
12 views

How to assess “reliability” for a non-stationary time series compared to a reference

I have two arterial pressure measurement devices, one of which is the reference. My goal is to assess reliability compared to the reference and the variables (demographics, passing time, etc.) ...
0
votes
0answers
33 views

Using the Standard Error of Prediction in the presence of practice effects

I’m wondering about the following hypothetical scenario. There’s a student who previously scored 40% on an examination with a pass mark of 50%, a mean mark of 60%, SD of 10 and a reliability of 0.6. ...
0
votes
0answers
5 views

How to Compute Krippendorff's Alpha for Text Continua?

I have a question about the inter-annotator reliability measure of Krippendorff's alpha for text continua or text unitization. The measure is explained in detail in the paper. I have texts and 2 ...
0
votes
1answer
18 views

How to calculate intra-observer error and average difference?

I am trying to assess intra-observer error in a setting where measurements where done repeatedly by a single observer. So far I'm having some conceptual and technical issues. In the dataset we have ...
0
votes
0answers
18 views

Krippendorff's Alpha for unitizing texts

I have a question about using alpha to measure Krippendorff's inter-annotator reliability in qualitative data. After reading the papers, I'm still not sure how to handle this case. I'm working on a ...
0
votes
1answer
19 views

Reliability Testing, Determining Weibull parameters

I want to use a Weibull analysis for demonstrating the reliability of a piece of equipment that will be set to cycle. How do I select the proper scale and shape parameters for the distribution if I ...
0
votes
0answers
10 views

first factor saturation vs. general factor saturation

a simple explanation between these two in the context of reliability analysis, specifically Cronbach's alpha.
1
vote
1answer
26 views

Unidimensionality in IRT

I recently came across the definition of unidimensionality (in the context of IRT and internal consistency reliability, specifically Cronbach's alpha). Johnsosn (2007) states that There is a one-...
0
votes
0answers
15 views

Cronbach's alpha interpretation - dichotomous data

I have calculated Cronbach's alpha in R using the psych package and I am trying to interpret them now. I want to properly understand this, as it is part of my ...
1
vote
1answer
31 views

Correlations - multiple observations & multiple observers

I haven't found quite this question on here. I have a data set that typically requires generalizability theory to untangle however I now face a question I believe g theory can not answer. 17 ...
0
votes
0answers
4 views

MTBF for a single failure

I have a machine for which I want to calculate the MTBF over a limited period of time. The failure rate is small and not constant. That is, I may encounter a single failure (or no failures) over a ...
3
votes
1answer
59 views

Prediction interval for a Weibull distribution

Suppose I am producing units and have tested the failure time of some proportion of them (say 10%), possibly with some right censoring if they didn't fail within the testing period. I'm able to fit a ...
0
votes
1answer
46 views

item-total correlation vs. inter-item correlation

I am assesing the reliability and validity of an 16-item multuiple choice test supposed to test language skills. Cronbach's apha is "questionable" with .65. I am having troubles interpreting the ...
1
vote
0answers
10 views

Measuring equivalence/reliability (ICC and consorts)

For a reliability study I’m administering patient reported outcomes (questionnaires) with 2 modes of administration (on paper and via an electronic device). I’d like to measure equivalence/reliability ...
0
votes
0answers
91 views

Difference between Cronbach's alpha and Pearson's coefficient

I am creating an indicator for social development which includes variables of health, education and economy. Since I have many variables, I decided to remove some of them based on dominion expertise ...
0
votes
0answers
10 views

Inter-rater reliability many coders but only one case

I have a specific performance assessment metric which I aim to get some reliability and validity data for. And I really would appreciate some help for the reliability: Background In my research, ...
1
vote
1answer
59 views

Item response theory for continuous variables, and estimating standard error of measurement

I really like how traditional item response theory (IRT) packages tell you the standard error of measurement conditional on one's ability level, and from that, you can calculate the test information ...
0
votes
0answers
26 views

How to calculate overall reliability for an IRT test with subtests

I have an IRT, 2 parameter model ability test with three subtests - verbal, numerical and abstract reasoning. I have been able to calculate the reliability of each subtest using the methods outlined ...
2
votes
1answer
34 views

Proof of direct proportionality between hazard rate function and probability density function

I'm reading up on reliability and I came across this question: Show that if the hazard function is decreasing, the PDF, $f(t)$, is also a decreasing function and its mode must therefore occur at t =...
0
votes
0answers
24 views

Generating Failure / Suspension data from Weibull Distribution

I've been trying to make some synthetic failure data in MATLAB and I'm coming across an issue when I try to refit the Weibull distribution from my synthetic data. For example, assuming a Weibull ...
0
votes
1answer
13 views

Reliability analysis for different measurement scales?

I'm new here :) I'm conducting a survey questionnaire which gathers feedback on English-as Second Language (ESL). I have two measurement scales in my survey. Questions on how much time they spend ...
0
votes
0answers
35 views

Coefficient omega: appropriate extraction of factors

I use omega_hiearchical to determine the internal consistency of a scale. If I understood it correctly, it tries to extract three lower level factors by default. But you can specify that you want to ...
1
vote
0answers
14 views

Model building, how to include information on data point quality?

I have a dataset with count data predictors and a marker on their reliability or completeness. With 2 options: 0 - Data is not confirmed (could be higher); 1 - data is confirmed to be complete. The ...
0
votes
0answers
9 views

Best way to measure reliability of auto-correlated data

I'm trying to compare different ambulatory devices for measuring heart rate against a gold standard in-lab assessment. What is the best way to determine whether a device is worth using. I can use ...
0
votes
0answers
17 views

How much time is needed between measures for test-retest reliability?

I am planning a meta-analysis on reliability of various loneliness scales. We want to determine beforehand what time frame between measures we will consider. For instance, if the time between 2 ...
1
vote
1answer
117 views

Connection between Hazard Ratios, Survival, and Probability

According to this wiki link, the hazard ratio relates to survival according to the following equation $$ S_1(t)=S_0(t)^r \quad (1) $$ where $S$ is the survival and $r$ is the hazard ratio. So from ...
0
votes
0answers
17 views

Evidence fusion-Dempster Shafer

I have developed four models that estimate the same state-traffic volume in 100 streets during a specific time period. I have actual volume for 25 streets to estimate compute the reliability of each ...
0
votes
0answers
33 views

Can VAR models capture all frequency interactions reliably and to the same extent?

I would like to assess how a set of ~50-100 time-series variables causally interact with each other. To do this, I am fitting a VAR model and using cross-validation to estimate model-order. After the ...
0
votes
1answer
31 views

In scale adaptation which should be done first? CFA or Reliability Analysis

I am adapting questionnaire cross-cultural. Questinonnaire has 11 item and two factor. I want to confirm the structure 1 item's corrected item total correlation is <.20. Should this item be ...
0
votes
0answers
14 views

Measurement theory and sample size calculation for multivariate testing

Let $\mathbf{Y}$ be a vector of independent, normally-distributed random variables. Let $S_1$, $S_2$ and $S_3$ be three non-overlapping samples of sizes $N_1$, $N_2$ and $N_3$, respectively. Let $M_A$ ...
1
vote
0answers
12 views

Is it correct to a total Cronbach´s alpha analysis to a tridimensional scale?

There´s a statement that says that Cronbach´s alpha assumes unidimensionality. In that sense, is it correct to calculate a global alpha? I mean, by including all 3 dimensions, and not by separating ...
1
vote
0answers
52 views

Can we statistically predict the next Lion Air incident? [closed]

Plugging the Lion Air incidents on Aviation Safety into a Crow/AMSAA tool gives MTBF of about 15 months at the moment. Is this reasonable for their fleet size of 115? Link to the: incident data ...
1
vote
0answers
17 views

Reliability of Composite Variable Made of 4 Measures?

I am conducting some analyses with a composite variable constructed by averaging 4 separate measures. I would like to know the reliability of the composite variable. Is Cronbach's alpha appropriate in ...
0
votes
1answer
95 views

Hazard function for a growing population

I want to model the expected failure rate in a growing population. Each individual has a high infant mortality (e.g. Weibull(shape=0.5, scale=100)). So the ...
1
vote
2answers
63 views

Book reference request for statistical reliability theory

I'm about to learn statistical reliability theory for the first time. I'm told that Barlow and Proschan's book entitled "Statistical theory of reliability and life testing" is a classic on this topic. ...
1
vote
0answers
109 views

Best easy and straightforward way of computing McDonald's omega in a unidimensional test with 'psych' in R

I'm preparing a masterclass in Psychometrics for a course, and I would like to introduce McDonald's omega coefficient as the proper index for assessing the reliability of a scale. For what I can see, ...
1
vote
0answers
431 views

McDonald's Omega: Assumptions, Coefficients and Interpretation

After browsing cross validated and several other sources on the web, I still cannot get a grip on McDonald's Omega as a measure of internal consistency. I have a hunch that many fellow social ...
1
vote
0answers
33 views

Index of within-participant consistency in two-alternative forced-choice (2AFC) task?

My study consists of 2AFC trials: Participants choose which one of a pair of stimuli has a certain quality. There are 9 pairs and 6 stimuli (2 categories of 3 stimuli each) in total. The stimuli ...
0
votes
1answer
31 views

How to calculate Mean Sample-Based Reliability Estimates Used for Analyses

I am conducting a meta analysis of Pearson's r , of correlations presented between test scores and job performance ratings. I need to present an uncorrected effect size and an effect size corrected ...
0
votes
1answer
155 views

Cronbach alpha with knowledge items?

I have a very specific question about the application of Cronbach's alpha to data with trichotomous responses (true-false-don't know). I have an acquaintance I'm helping replicate a study using a 45 ...
1
vote
1answer
37 views

Intraclass Correlations for Intraobserver-Reliability in Hierarchical Data

We have a dataset where multiple measurements of the same variable are taken from individuals and we want to measure the intraobserver reliability. However, the individuals each come from different ...
0
votes
0answers
139 views

Test-Retest reliability for mixed models

We have an experimental setup as follows: A set of subjects each repeat a single task multiple times in a single session, yielding a set of measurements. The same subjects then repeat the same task ...
0
votes
0answers
16 views

Cronbach's Alpha to Rate Videos

I would like to see if we can have reliable metrics that can be used to rate videos (V1:V15). I have 20 people for each video (id1:id20). And 2 metrics: ...
0
votes
0answers
76 views

Cronbach's alpha and yes-no-don't know scales

I am helping a friend analyze some questionnaire data, and I am currently a bit confused about how I should go about doing reliability analysis on her items. She is using a 45 item scale, assessing ...
0
votes
0answers
35 views

Reliability calculation using R

I have this handout about reliability calculation, but I just can't figure out how I could do this using R. Here is the simulated data. It is a questionaire that has ten quaestions and was answered ...
0
votes
0answers
17 views

Seeking a measure of multiple rater consistency for cateorical ratings with sparse data

I have multiple raters and sparse data. The raters can decide how many attributes apply to a person. I have many empty cells due to many attributes and few attributes actually chosen. What tools ...