Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

Filter by
Sorted by
Tagged with
0 votes
1 answer
29 views

What I supposed to do if Cronbach's alpha is negative?

I have 100 sample with 8 questions with answers between Yes or No (Guttman scale). Cronbach's alpha is badly negative. I have checked everything I could think of, but the result did not change. I have ...
user avatar
  • 1
0 votes
0 answers
10 views

SPSS - How can I measure whether a questionnaire consistently decomposes along similar dimensions? [closed]

I've been using a 14-item questionnaire that measures thought content. Each row in my dataset represents a completed questionnaire, showing the scores on each item. I've been decomposing sets of ...
user avatar
1 vote
2 answers
32 views

Calculating Cohen's Kappa in SPSS for a systematic review

I'm sure there's a simple answer to this but I haven't been able to find it yet. All the explanations I've found to calculate Cohen's Kappa in SPSS use data that is formatted with cases that are ...
user avatar
  • 13
1 vote
0 answers
17 views

Confirming that the failure rate of a new component is less than or equal to the failure rate of the old component with 95% confidence

I have a new component that is meant to replace an old component that has a well-documented failure rate. Both components are buttons, and the failure rate corresponds to the mean number of button ...
user avatar
1 vote
0 answers
24 views

Why is internal consistency a measure of reliability?

Why is 'internal consistency' a measure of reliability? It looks like a measure of validity instead. Definitions: "Internal consistency reflects the extent to which items within an instrument ...
user avatar
  • 93
0 votes
0 answers
8 views

SMALL SAMPLE SIZE: Can i use it for a validation study? What evidence supports this?

I would like to validate submaximal exercise tests; I will correlate the predicted maximum oxygen consumption values using the field tests (values will be derived from standard equation) with the ...
user avatar
  • 1
1 vote
1 answer
26 views

Lifetime estimation - Weibull vs Survival

Survival analysis is mainly about estimating the lifetime of nearly anything or even anything: And Weibull analysis is also about estimating the lifetime: How are they different (in general) and ...
user avatar
  • 2,197
0 votes
0 answers
13 views

psych package: reliability, should complete cases be used?

Would like to seek advice on instrument validation and the psych package in r. I would like to evaluate the reliability first (want to replicate the study here: https://doi.org/10.1186/s12885-018-5135-...
user avatar
  • 165
0 votes
0 answers
17 views

How to determine the lifetime?

I want to determine the average expected lifetime of devices of which I have data. This data comprises ~4500 generators of which ~3800 died and the others are still alive. Furthermore, I have ...
user avatar
  • 2,197
0 votes
1 answer
18 views

Identify / separate sorts of failure

I ran a Weibull analysis on data of devices which consists mainly of failures over time (80 % died, 20 % are still alive). In total there are around ~4500 devices. After conducting this analysis I ...
user avatar
  • 2,197
1 vote
0 answers
15 views

Which intraclass correlation coefficient (ICC) to use to verify the test-retest reliability of a questionnaire?

I want to check the test-retest reliability of a questionnaire. -A total of 300 adolescents from the same school (10 classes) participated in the study. -The instrument was applied in two moments with ...
user avatar
1 vote
0 answers
35 views

Finding large difference between Omega total and omega hierarchical - what does it mean?

I am computing reliability for some scales using the omega function in the R package psych. For one 3-item scale, the omega hierarchical is very close to 0 (.03), but omega total is reasonable (.79). ...
user avatar
  • 11
0 votes
1 answer
70 views

Test-retest reliability for a test taken twice but with a variable delay between tests

I have a dataset that is composed of a few thousand datapoints, each of which consists of two test results for a participant that were collected when they underwent the test at Time A and Time B. The ...
user avatar
0 votes
0 answers
12 views

Absolute reliability measure for interval data that are positive and negative

In biomechanics and physics, we often deal with vector data where values have a magnitude and direction (e.g. force, acceleration, velocity). The direction is indicated by the value being either + or -...
user avatar
1 vote
0 answers
41 views

Weibull and Lognormal Fits

I have simulated failure time data for some components. I tested whether a Weibull or lognormal distribution best fit the data. Please see Figures below. The figures provide median and $95\%$ ...
user avatar
  • 883
0 votes
0 answers
22 views

best use of survey data

So - I am trying to decide which resident to promote. Using their "Performance" as one metric (about 50% of their entire score) My idea was to normalize their survey score and then use that ...
user avatar
1 vote
0 answers
6 views

Reliable Change Index Question

I was wondering if someone could help me with a reliable change index question. I am looking to do pre/post analysis at the individual participant level. Is this method only appropriate for measures ...
user avatar
0 votes
0 answers
43 views

Warning about factor score estimation method when using omega() from psych

I am a beginner in statistics with R. I am currently analyzing the results of a questionnaire in my PhD research. I am trying to measure reliability using McDonald's omega rather than Cronbach's alpha....
user avatar
2 votes
1 answer
37 views

What are typical approaches to express the reliability of a categorical scale?

I have categorical scales consisting of items with each three answer options: 1) very much; 2) somewhat; 3) never. What would be a suitable approach to express reliabilities for scales like this? I am ...
user avatar
  • 469
0 votes
1 answer
24 views

Relationship between SD of MC test scores and test reliability?

The scores on a 80 question multiple-choice test had a mean 70% correct and standard deviation (SD) of 10 (SD of % correct). This SD is 2.5 times higher than the theoretical SD of 4 (based on N, p ...
user avatar
  • 3,026
0 votes
0 answers
19 views

Statistical test for questionnaire with various components

I was asked to analyze a research questionnaire that consists of a Likert scale, a checklist, and questions with yes, no, and don't know as response, as part of the pilot testing. I was struggling ...
user avatar
  • 51
0 votes
0 answers
20 views

residualised difference scores

I have read this paper and am interested in using the residualised difference scores to compute test-retest reliability of a Stroop task, yet I am a bit unsure about the procedure. Basically from what ...
user avatar
  • 452
0 votes
0 answers
12 views

Test-retest reliability with varying number of measurements

I developed a new method to measure a certain quantity on a radiological image of a patient. I have available varying numbers of studies per patient. I applied the method to all images available for ...
user avatar
0 votes
0 answers
25 views

How can I calculate the standard error of measurement (SEM) for crossed data that were repeated once?

I have acquired the responses of 10 persons that rated 15 items twice. All persons rated all 15 items. (10x15, rows=participants and columns=items). I have calculated the test retest reliability of ...
user avatar
0 votes
0 answers
30 views

Tests for item rating reliability (Cronbach's alpha?)

Here is the situation: I am asking participants to provide some names of a certain category. (Imagine a verbal fluency task where you are asked to name as many animals as you can in the space of 1 ...
user avatar
0 votes
0 answers
6 views

Calculating interrater reliability for 6 raters in a large sample, what proportion to dual code?

I am conducting a study where clinical notes are coded to produce categorical data variables (6 variables with 3-5 responses options). I have 6 coders splitting 2000 participants between them. This is ...
user avatar
1 vote
0 answers
19 views

How to conduct a meta-analysis on ICCs (i.e., test-retest reliability)?

I have calculated two-way single consistency ICCs in R using the irr package. There are multiple ICCs representing test-retest reliability across these time points. Is there a way to compute a meta-...
user avatar
  • 309
0 votes
1 answer
58 views

95 percentile for failure frequency with Poisson process

I am trying to recreate the stats for 'failure frequency' - in this case the 95 percentile - shown in the attached images below. For one set of failure data there are 27 failures over a cumulative ...
user avatar
0 votes
0 answers
69 views

Comparing test-retest reliability between methods by using Pearson Correlation Coefficients

For my research I wanted to compare two survey answering methods, specifically to see which would be me more reliable. So I conducted two surveys, one for each method, then conducted those same ...
user avatar
3 votes
1 answer
104 views

Hazard function of a simple system of components

I have been asked to complete these tasks, and a few more, but I am stuck somehow: Figure 6 is the simple system above. For task number 1: [DONE] I started by recognizing that the total survival time ...
user avatar
  • 33
0 votes
0 answers
98 views

Alternative to Intra-class correlation coefficient (ICC) for continuous data

I am working on a research project that involves a reliability study. In this case, 3 raters have assigned two sets of scores to 50 items: an ordinal score on a scale from 1 - 3, and a continuous ...
user avatar
1 vote
1 answer
25 views

Statistical test on a feature that changes over time

Let me explain my use case with an example. Let's say some phone manufacturer such as apple iphone monitors hardware failure rates. I have a hypothesis that says the latest ios version is resulting in ...
user avatar
0 votes
0 answers
11 views

Intrarater reliability with three tests on two groups?

i have a question regarding my upcoming study for my master's thesis where the focus is on Intrarater Reliability and Validity. The study looks as follows: Two groups (Group 1 and Group 2) Each Group ...
user avatar
0 votes
0 answers
23 views

What measure of agreeement should I use for time series data?

I am trying to compare the agreement between two measurements. Each measurement tells me when a device is worn or not worn (binary) and is sampled at the same frequency (1Hz). The measurements were ...
user avatar
1 vote
0 answers
29 views

What Heterogeneity/Between-Study Variance Estimator to use in a Reliability Meta-Analysis/Reliability Generalization Random Effects Model?

This is an extension of this question but for reliability generalization studies. Could anyone recommend a between-study variance / heterogeneity () estimator and method for computing confidence ...
user avatar
0 votes
1 answer
74 views

Understanding the probability distributions behind a Monte Carlo experiment

A colleague and I are trying to model the expected maintenance cost/h (E[C/h]) of a component A on an aircraft over its life based on its reliability distribution. As the component fail, it's replaced ...
user avatar
1 vote
1 answer
57 views

Reliability Meta-Analysis/Reliability Generalization: Mixing Methods for Calculating Transformations, Sampling Variances, and Weights

I am currently planning a reliability meta-analysis or reliability generalization study (Vacha-Haase, 1998). I write this question assuming a random-effects model has been selected and Cronbach's a is ...
user avatar
1 vote
1 answer
27 views

Defective Subpopulation Distribution and Conditional Probability

A colleague and I have tried two different approaches to this problem, both of which seem to make sense but are resulting in very different answers. Suppose we have some units undergoing B hours of ...
user avatar
0 votes
0 answers
15 views

Testing for reliability of new machine

We have a new lab machine A that produces results (positive/negatives). We would like to understand if the machine is properly set up (producing results as manufacturer estimation), and whether it ...
user avatar
0 votes
0 answers
47 views

How to compare the test-retest reliability of two groups?

I have one group (NBD, N=23) and another group (PWA, N=22) who each have data at Test (time point 1) and retest (time point 2). I've calculated test-retest reliability using a oneway, consistency, ...
user avatar
2 votes
1 answer
35 views

Compare the bootstrap results of 2 datasets with split-half reliability

I am currently working on the analysis of a metric comparing two methods (A & B) for subjective annotation. The two methods give 2 types of data which after processing gives scores relatives to ...
user avatar
  • 21
0 votes
0 answers
35 views

How to determine number of boost for calculating Krippendorff’s Alpha?

I'm using the SPSS macro for computing Krippendorff’s Alpha, developed by Andrew F. Hayes & Klaus Krippendorff. One argument needed is the units of bootstrapping, and I did not find formula for ...
user avatar
0 votes
0 answers
14 views

Is it possible to calculate the correlation coefficients between subscales using item reliability statistics?

I need to find the correlation coefficients between each of the subscales of the SRS scale. I found this article that measures item reliability and cronbachs alpha for the subscales. How can I get the ...
user avatar
2 votes
0 answers
327 views

How to calculate two-way mixed ICC?

I want to calculate ICC for two-way mixed model, single rater, and either for consistency and absolute agreement. I found two packages for calculating ICC (intra-class correlation coefficient), that ...
user avatar
1 vote
1 answer
51 views

Weibull Reliability Test Demonstration

I'll use the following example to help frame my question. The example comes from example 5 in https://www.weibull.com/pubs/2015_RAMS_right_sample_size.pdf Example 5: Let's assume that the failure ...
user avatar
  • 163
0 votes
1 answer
73 views

Right censored data

I have some data that is right censored. A study has recorded data for several years and some units have failed and some units have not failed (right censored). Now, some of the units observe a minor ...
user avatar
  • 883
1 vote
0 answers
11 views

Can Splithalf Reliability Be Caclulated for an Aggregate Measure?

I have been asked by a journal editor to provide the splithalf reliability for an aggregate dependent variable, d', which is a Signal Detection parameter that captures perceptual sensitivity. However, ...
user avatar
  • 11
0 votes
0 answers
50 views

Interpreting low Cronbach's alpha after PCA

I've been running an exploratory analyses on my dataset. I have completed PCA analysis and decided to retain 3 factors. In the subsequent reliability analysis, I discovered that the overall ...
user avatar
0 votes
0 answers
32 views

Confusion matrix test-retest reliability (McNemar or paired t-test)

I would like to ensure test retest reliability between test runs. Each run produces a 2x2 confusion matrix reporting the number of true positives, false positives, false negatives, and true negatives. ...
user avatar
0 votes
0 answers
23 views

How can I assess within-subject reliability of Bayesian models?

I already posted this question in another forum, but didn’t reach a lot of people (with Bayesian expertise). Hopefully, someone in here can help me out: I would like to do a Bayesian analysis on the ...
user avatar

1
2 3 4 5
10