# Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

492 questions
Filter by
Sorted by
Tagged with
29 views

### What I supposed to do if Cronbach's alpha is negative?

I have 100 sample with 8 questions with answers between Yes or No (Guttman scale). Cronbach's alpha is badly negative. I have checked everything I could think of, but the result did not change. I have ...
10 views

### SPSS - How can I measure whether a questionnaire consistently decomposes along similar dimensions? [closed]

I've been using a 14-item questionnaire that measures thought content. Each row in my dataset represents a completed questionnaire, showing the scores on each item. I've been decomposing sets of ...
1 vote
32 views

### Calculating Cohen's Kappa in SPSS for a systematic review

I'm sure there's a simple answer to this but I haven't been able to find it yet. All the explanations I've found to calculate Cohen's Kappa in SPSS use data that is formatted with cases that are ...
• 13
1 vote
17 views

### Confirming that the failure rate of a new component is less than or equal to the failure rate of the old component with 95% confidence

I have a new component that is meant to replace an old component that has a well-documented failure rate. Both components are buttons, and the failure rate corresponds to the mean number of button ...
1 vote
24 views

### Why is internal consistency a measure of reliability?

Why is 'internal consistency' a measure of reliability? It looks like a measure of validity instead. Definitions: "Internal consistency reflects the extent to which items within an instrument ...
• 93
8 views

### SMALL SAMPLE SIZE: Can i use it for a validation study? What evidence supports this?

I would like to validate submaximal exercise tests; I will correlate the predicted maximum oxygen consumption values using the field tests (values will be derived from standard equation) with the ...
• 1
1 vote
26 views

### Lifetime estimation - Weibull vs Survival

Survival analysis is mainly about estimating the lifetime of nearly anything or even anything: And Weibull analysis is also about estimating the lifetime: How are they different (in general) and ...
• 2,197
13 views

### psych package: reliability, should complete cases be used?

Would like to seek advice on instrument validation and the psych package in r. I would like to evaluate the reliability first (want to replicate the study here: https://doi.org/10.1186/s12885-018-5135-...
• 165
17 views

### How to determine the lifetime?

I want to determine the average expected lifetime of devices of which I have data. This data comprises ~4500 generators of which ~3800 died and the others are still alive. Furthermore, I have ...
• 2,197
18 views

### Identify / separate sorts of failure

I ran a Weibull analysis on data of devices which consists mainly of failures over time (80 % died, 20 % are still alive). In total there are around ~4500 devices. After conducting this analysis I ...
• 2,197
1 vote
15 views

### Which intraclass correlation coefficient (ICC) to use to verify the test-retest reliability of a questionnaire?

I want to check the test-retest reliability of a questionnaire. -A total of 300 adolescents from the same school (10 classes) participated in the study. -The instrument was applied in two moments with ...
• 61
1 vote
35 views

### Finding large difference between Omega total and omega hierarchical - what does it mean?

I am computing reliability for some scales using the omega function in the R package psych. For one 3-item scale, the omega hierarchical is very close to 0 (.03), but omega total is reasonable (.79). ...
• 11
70 views

### Test-retest reliability for a test taken twice but with a variable delay between tests

I have a dataset that is composed of a few thousand datapoints, each of which consists of two test results for a participant that were collected when they underwent the test at Time A and Time B. The ...
12 views

### Absolute reliability measure for interval data that are positive and negative

In biomechanics and physics, we often deal with vector data where values have a magnitude and direction (e.g. force, acceleration, velocity). The direction is indicated by the value being either + or -...
1 vote
41 views

### Weibull and Lognormal Fits

I have simulated failure time data for some components. I tested whether a Weibull or lognormal distribution best fit the data. Please see Figures below. The figures provide median and $95\%$ ...
• 883
22 views

### best use of survey data

So - I am trying to decide which resident to promote. Using their "Performance" as one metric (about 50% of their entire score) My idea was to normalize their survey score and then use that ...
• 11
1 vote
6 views

### Reliable Change Index Question

I was wondering if someone could help me with a reliable change index question. I am looking to do pre/post analysis at the individual participant level. Is this method only appropriate for measures ...
43 views

### Warning about factor score estimation method when using omega() from psych

I am a beginner in statistics with R. I am currently analyzing the results of a questionnaire in my PhD research. I am trying to measure reliability using McDonald's omega rather than Cronbach's alpha....
37 views

### What are typical approaches to express the reliability of a categorical scale?

I have categorical scales consisting of items with each three answer options: 1) very much; 2) somewhat; 3) never. What would be a suitable approach to express reliabilities for scales like this? I am ...
• 469
24 views

### Relationship between SD of MC test scores and test reliability?

The scores on a 80 question multiple-choice test had a mean 70% correct and standard deviation (SD) of 10 (SD of % correct). This SD is 2.5 times higher than the theoretical SD of 4 (based on N, p ...
• 3,026
19 views

### Statistical test for questionnaire with various components

I was asked to analyze a research questionnaire that consists of a Likert scale, a checklist, and questions with yes, no, and don't know as response, as part of the pilot testing. I was struggling ...
• 51
20 views

### residualised difference scores

I have read this paper and am interested in using the residualised difference scores to compute test-retest reliability of a Stroop task, yet I am a bit unsure about the procedure. Basically from what ...
• 452
12 views

### Test-retest reliability with varying number of measurements

I developed a new method to measure a certain quantity on a radiological image of a patient. I have available varying numbers of studies per patient. I applied the method to all images available for ...
25 views

### How can I calculate the standard error of measurement (SEM) for crossed data that were repeated once?

I have acquired the responses of 10 persons that rated 15 items twice. All persons rated all 15 items. (10x15, rows=participants and columns=items). I have calculated the test retest reliability of ...
• 13
30 views

### Tests for item rating reliability (Cronbach's alpha?)

Here is the situation: I am asking participants to provide some names of a certain category. (Imagine a verbal fluency task where you are asked to name as many animals as you can in the space of 1 ...
6 views

### Calculating interrater reliability for 6 raters in a large sample, what proportion to dual code?

I am conducting a study where clinical notes are coded to produce categorical data variables (6 variables with 3-5 responses options). I have 6 coders splitting 2000 participants between them. This is ...
1 vote
19 views

### How to conduct a meta-analysis on ICCs (i.e., test-retest reliability)?

I have calculated two-way single consistency ICCs in R using the irr package. There are multiple ICCs representing test-retest reliability across these time points. Is there a way to compute a meta-...
• 309
58 views

### 95 percentile for failure frequency with Poisson process

I am trying to recreate the stats for 'failure frequency' - in this case the 95 percentile - shown in the attached images below. For one set of failure data there are 27 failures over a cumulative ...
69 views

### Comparing test-retest reliability between methods by using Pearson Correlation Coefficients

For my research I wanted to compare two survey answering methods, specifically to see which would be me more reliable. So I conducted two surveys, one for each method, then conducted those same ...
104 views

### Hazard function of a simple system of components

I have been asked to complete these tasks, and a few more, but I am stuck somehow: Figure 6 is the simple system above. For task number 1: [DONE] I started by recognizing that the total survival time ...
• 33
98 views

### Alternative to Intra-class correlation coefficient (ICC) for continuous data

I am working on a research project that involves a reliability study. In this case, 3 raters have assigned two sets of scores to 50 items: an ordinal score on a scale from 1 - 3, and a continuous ...
1 vote
25 views

### Statistical test on a feature that changes over time

Let me explain my use case with an example. Let's say some phone manufacturer such as apple iphone monitors hardware failure rates. I have a hypothesis that says the latest ios version is resulting in ...
11 views

### Intrarater reliability with three tests on two groups?

i have a question regarding my upcoming study for my master's thesis where the focus is on Intrarater Reliability and Validity. The study looks as follows: Two groups (Group 1 and Group 2) Each Group ...
23 views

### What measure of agreeement should I use for time series data?

I am trying to compare the agreement between two measurements. Each measurement tells me when a device is worn or not worn (binary) and is sampled at the same frequency (1Hz). The measurements were ...
1 vote
29 views

### What Heterogeneity/Between-Study Variance Estimator to use in a Reliability Meta-Analysis/Reliability Generalization Random Effects Model?

This is an extension of this question but for reliability generalization studies. Could anyone recommend a between-study variance / heterogeneity () estimator and method for computing confidence ...
• 479
74 views

### Understanding the probability distributions behind a Monte Carlo experiment

A colleague and I are trying to model the expected maintenance cost/h (E[C/h]) of a component A on an aircraft over its life based on its reliability distribution. As the component fail, it's replaced ...
• 151
1 vote
57 views

### Reliability Meta-Analysis/Reliability Generalization: Mixing Methods for Calculating Transformations, Sampling Variances, and Weights

I am currently planning a reliability meta-analysis or reliability generalization study (Vacha-Haase, 1998). I write this question assuming a random-effects model has been selected and Cronbach's a is ...
• 479
1 vote
27 views

### Defective Subpopulation Distribution and Conditional Probability

A colleague and I have tried two different approaches to this problem, both of which seem to make sense but are resulting in very different answers. Suppose we have some units undergoing B hours of ...
15 views

### Testing for reliability of new machine

We have a new lab machine A that produces results (positive/negatives). We would like to understand if the machine is properly set up (producing results as manufacturer estimation), and whether it ...
47 views

### How to compare the test-retest reliability of two groups?

I have one group (NBD, N=23) and another group (PWA, N=22) who each have data at Test (time point 1) and retest (time point 2). I've calculated test-retest reliability using a oneway, consistency, ...
35 views

### Compare the bootstrap results of 2 datasets with split-half reliability

I am currently working on the analysis of a metric comparing two methods (A & B) for subjective annotation. The two methods give 2 types of data which after processing gives scores relatives to ...
• 21
35 views

### How to determine number of boost for calculating Krippendorff’s Alpha?

I'm using the SPSS macro for computing Krippendorff’s Alpha, developed by Andrew F. Hayes & Klaus Krippendorff. One argument needed is the units of bootstrapping, and I did not find formula for ...
• 39
14 views

### Is it possible to calculate the correlation coefficients between subscales using item reliability statistics?

I need to find the correlation coefficients between each of the subscales of the SRS scale. I found this article that measures item reliability and cronbachs alpha for the subscales. How can I get the ...
327 views

### How to calculate two-way mixed ICC?

I want to calculate ICC for two-way mixed model, single rater, and either for consistency and absolute agreement. I found two packages for calculating ICC (intra-class correlation coefficient), that ...
• 39
1 vote
51 views

### Weibull Reliability Test Demonstration

I'll use the following example to help frame my question. The example comes from example 5 in https://www.weibull.com/pubs/2015_RAMS_right_sample_size.pdf Example 5: Let's assume that the failure ...
• 163
73 views

### Right censored data

I have some data that is right censored. A study has recorded data for several years and some units have failed and some units have not failed (right censored). Now, some of the units observe a minor ...
• 883
1 vote
11 views

### Can Splithalf Reliability Be Caclulated for an Aggregate Measure?

I have been asked by a journal editor to provide the splithalf reliability for an aggregate dependent variable, d', which is a Signal Detection parameter that captures perceptual sensitivity. However, ...
• 11
50 views

### Interpreting low Cronbach's alpha after PCA

I've been running an exploratory analyses on my dataset. I have completed PCA analysis and decided to retain 3 factors. In the subsequent reliability analysis, I discovered that the overall ...
• 11