Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

Filter by
Sorted by
Tagged with
2 votes
1 answer
22 views

Do you have to assess reliability for non-composite scores?

I feel like this is probably a really stupid question but I'm having issues finding the answer. When I think of reliability, normally I think of several items on a test that are compiled into some ...
0 votes
0 answers
73 views

The GMM estimator is reliable under the given conditions

Consider the linear model $$y_t= x_t'\beta +\epsilon_t$$ for $t=1,...,T$. where $x_t= ( x_{1t} \ \ x_{2t} \ \ ... \ \ x_{kt}) $ and $\beta$ is $(k\times 1)$ vector of unknown coefficients. Given $z_t= ...
  • 976
4 votes
2 answers
36 views

Does using a Likert Scale require me to always measure the internal consistency?

I am doing a Masters project (in environmental sociology) which includes a series of 15 questions that use a 6-point Likert scale to respond to various statements. In my pilot survey, my supervisor ...
  • 41
1 vote
0 answers
10 views

What are omega 1 and omega 2 in the reliability output from R?

I read this very useful article on omega reliability coefficients and have since been toying around with it in R (using the reliability function from the ...
0 votes
1 answer
17 views

How to calculate categorical omega reliability coefficient when some items only have one value?

From what I understand, running omega reliability in R is quite straightforward, even for binary responses. However, I've encountered a situation when running this command: ...
3 votes
0 answers
27 views

What are the rules of thumb for interpreting correlation coefficients corrected for unreliability?

I performed a series of meta-analyses in which we corrected correlation coefficients for measurement unreliability. A reviewer is asking us to define the magnitude of the meta-analytically-produced ...
  • 333
1 vote
0 answers
21 views

Reliability of opensource ML packages [closed]

While working on a machine learning project, in addition to the widely known opensource ML packages like tidyverse (R), pandas/scikitlearn (python), we end up using lot of other little known packages, ...
0 votes
0 answers
17 views

Can I summarize different forms of intraclass correlation coefficients (ICCs) descriptively?

There are various forms of intraclass correlation coefficients (ICCs) that differ in terms of their "model," "type," and "definition" (Koo & Li, 2016). So my question ...
1 vote
2 answers
67 views

Cronbach’s alpha not equal to ICC 3,k

I am calculating ICC over 16 items rated by 41 raters, using the ICC function from the psych package. The output is: ...
  • 11
1 vote
1 answer
19 views

Inter-rater reliability

In my research, I use a questionnaire so that students mention some classmates according to a group of characteristics. Each item is a different characteristic, and for each one they must name the ...
1 vote
0 answers
34 views

Computing IRT score reliability across a range of ability in mirt

I am using mirt and would like to get reliability estimates across a range of ability for a 1-factor scale. Essentially, I would like to extract the data used to ...
2 votes
0 answers
32 views

Composite reliability for a simple parallel model

I want to calculate the composite reliability for a simple parallel model as follow. And the below is what the textbook says (you can skip this part if you want) When X is a composite score, using A ...
  • 107
0 votes
0 answers
25 views

How do I calculate different reliabilty levels of theta when standard error of theta is known? (IRT-model) [duplicate]

I've recently seen a plot where the level of reliability is given for different SE values of theta estimates: I have also plotted the standard errors of theta as a function of theta but now I am ...
  • 31
1 vote
1 answer
38 views

Calculation of reliability of scale scores in an IRT- model

While trying to compute the reliability of scale scores obtained from IRT models (bifactor models and unidimensional models) I was wondering why there are so many published papers who report omega ...
  • 31
0 votes
0 answers
16 views

How to combine multiple reliability data with non-equal means and normal distribution?

Suppose I have a reliability data of an item from three sources (see picture). All of the sources provide reliability with different means ($\mu_1$ , $\mu_2$, $\mu_3$) and normal distribution ($\...
0 votes
1 answer
121 views

Negative values for Krippendorff 's alpha

I am trying to assess the inter-rater reliability for 5 raters about 36 questions. My variables are ordinal. Am I doing something wrong? Or what is the meaning of negative values?
1 vote
0 answers
10 views

What analysis is acceptable to assess reliability of a questionnarie composed for nominal and ordinal items?

I need to asses the reliability of a questionnarie composed by a series of n items that respond to n characteristics. In each item the surveyed idividuals have to nominate three individuals and assign ...
2 votes
1 answer
84 views

Log Cauchy distribution as lifetime distribution

Though I know that the moments of Log-Cauchy distribution do not exist, is it possible to use the Log-Cauchy distribution as a lifetime distribution under Type-II censoring? Because the MLE of the ...
  • 93
0 votes
1 answer
20 views

Are Cronbach's alpha of the subscale the same as in correlated-factor model and bifactor model?

I have an instrument that holds a correlated-factor model and a bifactor model (one general model and four orthogonal factors). I need to report the reliability of each subscale. I've computed Omega (...
0 votes
0 answers
28 views

What is an appropriate measure or test of reliability (internal consistency) for count data?

Internal consistency is often measured using Cronbach's Alpha which is a correlation based test. However, correlating count data (e.g. Item 1 = 4, Item 2 = 0, Item 3 = 3, etc.) with each other isn't ...
  • 381
1 vote
2 answers
151 views

Using remaining useful life predictions to assess model fit with right censored data

I am modelling time to failure of some units. All units are made my the same manufacturer and there are no recorded covariates to distinguish between the units. This means that the central, $95\%$ say,...
  • 823
4 votes
2 answers
116 views

Hazard function and survival analysis

I have a function $\lambda(t)$ which returns the instant probability of dying at a time $t$. I'd like to compute the function $F(t)$ which returns the probability of being dead at a certain time $t$. ...
0 votes
1 answer
38 views

What I supposed to do if Cronbach's alpha is negative? [closed]

I have 100 sample with 8 questions with answers between Yes or No (Guttman scale). Cronbach's alpha is badly negative. I have checked everything I could think of, but the result did not change. I have ...
  • 1
1 vote
2 answers
83 views

Calculating Cohen's Kappa in SPSS for a systematic review

I'm sure there's a simple answer to this but I haven't been able to find it yet. All the explanations I've found to calculate Cohen's Kappa in SPSS use data that is formatted with cases that are ...
  • 13
1 vote
0 answers
24 views

Confirming that the failure rate of a new component is less than or equal to the failure rate of the old component with 95% confidence

I have a new component that is meant to replace an old component that has a well-documented failure rate. Both components are buttons, and the failure rate corresponds to the mean number of button ...
1 vote
0 answers
42 views

Why is internal consistency a measure of reliability?

Why is 'internal consistency' a measure of reliability? It looks like a measure of validity instead. Definitions: "Internal consistency reflects the extent to which items within an instrument ...
  • 317
0 votes
0 answers
9 views

SMALL SAMPLE SIZE: Can i use it for a validation study? What evidence supports this?

I would like to validate submaximal exercise tests; I will correlate the predicted maximum oxygen consumption values using the field tests (values will be derived from standard equation) with the ...
  • 1
1 vote
1 answer
68 views

Lifetime estimation - Weibull vs Survival

Survival analysis is mainly about estimating the lifetime of nearly anything or even anything: And Weibull analysis is also about estimating the lifetime: How are they different (in general) and ...
  • 2,421
0 votes
0 answers
54 views

psych package: reliability, should complete cases be used?

Would like to seek advice on instrument validation and the psych package in r. I would like to evaluate the reliability first (want to replicate the study here: https://doi.org/10.1186/s12885-018-5135-...
  • 165
0 votes
0 answers
25 views

How to determine the lifetime?

I want to determine the average expected lifetime of devices of which I have data. This data comprises ~4500 generators of which ~3800 died and the others are still alive. Furthermore, I have ...
  • 2,421
0 votes
1 answer
19 views

Identify / separate sorts of failure

I ran a Weibull analysis on data of devices which consists mainly of failures over time (80 % died, 20 % are still alive). In total there are around ~4500 devices. After conducting this analysis I ...
  • 2,421
1 vote
0 answers
46 views

Which intraclass correlation coefficient (ICC) to use to verify the test-retest reliability of a questionnaire?

I want to check the test-retest reliability of a questionnaire. -A total of 300 adolescents from the same school (10 classes) participated in the study. -The instrument was applied in two moments with ...
1 vote
0 answers
134 views

Finding large difference between Omega total and omega hierarchical - what does it mean?

I am computing reliability for some scales using the omega function in the R package psych. For one 3-item scale, the omega hierarchical is very close to 0 (.03), but omega total is reasonable (.79). ...
  • 11
0 votes
1 answer
92 views

Test-retest reliability for a test taken twice but with a variable delay between tests

I have a dataset that is composed of a few thousand datapoints, each of which consists of two test results for a participant that were collected when they underwent the test at Time A and Time B. The ...
0 votes
0 answers
13 views

Absolute reliability measure for interval data that are positive and negative

In biomechanics and physics, we often deal with vector data where values have a magnitude and direction (e.g. force, acceleration, velocity). The direction is indicated by the value being either + or -...
1 vote
0 answers
75 views

Weibull and Lognormal Fits

I have simulated failure time data for some components. I tested whether a Weibull or lognormal distribution best fit the data. Please see Figures below. The figures provide median and $95\%$ ...
  • 823
0 votes
0 answers
22 views

best use of survey data

So - I am trying to decide which resident to promote. Using their "Performance" as one metric (about 50% of their entire score) My idea was to normalize their survey score and then use that ...
1 vote
0 answers
6 views

Reliable Change Index Question

I was wondering if someone could help me with a reliable change index question. I am looking to do pre/post analysis at the individual participant level. Is this method only appropriate for measures ...
0 votes
0 answers
95 views

Warning about factor score estimation method when using omega() from psych

I am a beginner in statistics with R. I am currently analyzing the results of a questionnaire in my PhD research. I am trying to measure reliability using McDonald's omega rather than Cronbach's alpha....
2 votes
1 answer
57 views

What are typical approaches to express the reliability of a categorical scale?

I have categorical scales consisting of items with each three answer options: 1) very much; 2) somewhat; 3) never. What would be a suitable approach to express reliabilities for scales like this? I am ...
  • 491
0 votes
1 answer
24 views

Relationship between SD of MC test scores and test reliability?

The scores on a 80 question multiple-choice test had a mean 70% correct and standard deviation (SD) of 10 (SD of % correct). This SD is 2.5 times higher than the theoretical SD of 4 (based on N, p ...
  • 3,086
0 votes
0 answers
21 views

Statistical test for questionnaire with various components

I was asked to analyze a research questionnaire that consists of a Likert scale, a checklist, and questions with yes, no, and don't know as response, as part of the pilot testing. I was struggling ...
  • 51
0 votes
0 answers
24 views

residualised difference scores

I have read this paper and am interested in using the residualised difference scores to compute test-retest reliability of a Stroop task, yet I am a bit unsure about the procedure. Basically from what ...
  • 442
0 votes
0 answers
17 views

Test-retest reliability with varying number of measurements

I developed a new method to measure a certain quantity on a radiological image of a patient. I have available varying numbers of studies per patient. I applied the method to all images available for ...
0 votes
0 answers
25 views

How can I calculate the standard error of measurement (SEM) for crossed data that were repeated once?

I have acquired the responses of 10 persons that rated 15 items twice. All persons rated all 15 items. (10x15, rows=participants and columns=items). I have calculated the test retest reliability of ...
0 votes
0 answers
38 views

Tests for item rating reliability (Cronbach's alpha?)

Here is the situation: I am asking participants to provide some names of a certain category. (Imagine a verbal fluency task where you are asked to name as many animals as you can in the space of 1 ...
0 votes
0 answers
12 views

Calculating interrater reliability for 6 raters in a large sample, what proportion to dual code?

I am conducting a study where clinical notes are coded to produce categorical data variables (6 variables with 3-5 responses options). I have 6 coders splitting 2000 participants between them. This is ...
1 vote
0 answers
22 views

How to conduct a meta-analysis on ICCs (i.e., test-retest reliability)?

I have calculated two-way single consistency ICCs in R using the irr package. There are multiple ICCs representing test-retest reliability across these time points. Is there a way to compute a meta-...
  • 381
0 votes
2 answers
130 views

95 percentile for failure frequency with Poisson process

I am trying to recreate the stats for 'failure frequency' - in this case the 95 percentile - shown in the attached images below. For one set of failure data there are 27 failures over a cumulative ...
0 votes
0 answers
121 views

Comparing test-retest reliability between methods by using Pearson Correlation Coefficients

For my research I wanted to compare two survey answering methods, specifically to see which would be me more reliable. So I conducted two surveys, one for each method, then conducted those same ...

1
2 3 4 5
11