Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

Using S statistic in place of Cohen's Kappa: R code provided [migrated]

This article proposes a new statistic called $S$ statistic to replace Cohen's Kappa for inter-rater agreement using categorical data. I know I can compute KAPPA for each category using the ...
0
votes
1answer
46 views

Calculating the Expected life of network with hazard functions

I have been asked to solve some questions in the statistical program R. First, I want to find the expected life length (by numeric integration) of a network with 3 parallel components (T1,T2,T3) and ...
0
votes
0answers
15 views

Finding stability in a time series of success/failure data — Mixed Model Random Effects

I have a very large data set of success/failure data for hundreds of subjects. The chance of success is rather small but does/should stabilize over time. Subjects have varying levels of observations/...
0
votes
0answers
8 views

Validity and Reliablity Multiple Choice Question Questionnaire

I gave questionnaires to 7-8-year-old children to answer questions about oral health knowledge. I've already collected the data but I'm still confused with validity and the reliability in SPSS. I ...
2
votes
1answer
36 views

What transformation can I use to enable a relatively unbiased meta-analysis of $\alpha$ reliability?

I want to meta-analyse reliability estimates. For simplicity, assume the reliability estimate in question is Cronbach's $\alpha$ (aka Guttman's $\lambda$-3). Various packages exist to do meta-analyses ...
4
votes
1answer
64 views

Difference between empirical and marginal reliability of an IRT model

I am using the mirt library in R to fit an instrument (binary responses) comprising two dimensions. In the mirt documentation are mentioned two types of reliability. I would like to ask what is the ...
0
votes
0answers
23 views

Generalizability study of subscaled rating instrument

I have a school rating ($1$ thru $4$) instrument that consists of $9$ subscales (e.g., classroom instruction, school management etc.). Under each subscale, I have $6$ items on which rating occurs. I ...
0
votes
0answers
10 views

Cronbachs alpha interpretation - what does var.r and med.r mean?

i have used the psych::alpha to find the cronbachs alpha values, but i cannot find any interpretation for some parts of the output, namely: ...
0
votes
1answer
20 views

Is it always needed to do a validity test for variables?

Let's say I have one variable called "perceived image", which is measured with 6 items (questions) that I adapted from previous literature. If I have a Cronbach's alpha > 0.7 (for reliability), do I ...
0
votes
0answers
22 views

How do I combine the categories of a classification system to find the maximum reliability?

I described my study design in another question. Briefly, I recruited 16 unique raters to classify 35 items according to a seven-category classification system. I measured the interrater reliability ...
2
votes
0answers
39 views

Correcting for reliability scores for Hedges g

I've received a revise and resubmit on my meta-analysis investigating the effects of a positive psychology intervention on depression and anxiety symptoms. I used Hedges' g as the effect size within a ...
0
votes
0answers
17 views

TTF and Relibility data form

Assume the car engine has 4 failure in one month. Example: Failure at day 10, 15, 23, 29. Is TTF maintained after each failure as shown in the following table, or it is increasing? How to calculate ...
0
votes
1answer
16 views

Which distribution or process should be used for wearout reliability modeling?

When modeling the reliability of a system, it is usual to use exponential distribution to model errors that occur randomly throughout the system's useful lifetime (the middle part of the well-known ...
0
votes
0answers
11 views

inter-observer reliability

I want to test the inter observer reliability between myself and one other observer on three different groups of data. I am unsure hat the best statistic is to go with the data I have collected: 1) ...
0
votes
0answers
29 views

Reliability estimation for questionnaire - is Cronbach's a applicable?

I am fairly new to scale analysis and I need to estimate the reliability and validity of a 5 item scale (5 questions, 2 have 4 choices abcd and 3 have 2 choices ab). The scale is supposed to be ...
0
votes
0answers
35 views

Minitab / Survival Analysis / Multiply Right Censor

I am doing an analysis to see if investments in leadership training have decreased attrition. The data I have is: Employee Start Date Hire Quarter Term Date (if not still employed) Status (Active ...
0
votes
0answers
38 views

Intra-class Correlation Coefficient (ICC) for a single subject

I need to calculate a score that reflects the level of disagreement/ambiguity between raters for a music clip. The experimental set-up is that 10 respondents listen to a clip with a one-dimensional ...
0
votes
0answers
15 views

Cohen's Kappa (observer variability) with partially ordered sets

I was wondering if there is a common way to compute Cohen's k when the values are partially ordered sets (of a lattice). My data roughly looks like the table below. There is a default lower bound (...
6
votes
0answers
61 views

Bootstrap Prediction Interval: which residuals to use and which method?

I ask this question referring to the post: Bootstrap prediction interval, where a step by step method for calculating the prediction interval for linear regression models is explained. In the ...
0
votes
1answer
40 views

Real-life application of min(X,Y) series system

Let $X$ and $Y$ be the random variables. What are the real life situations where $\min(X,Y)$ arises?
1
vote
1answer
46 views

Computing the accuracy of an answer

Let's assume I have three experts providing an answer. Expert 1 is 95% accurate (The likelihood of providing a correct answer is 95%). Expert 2 is 90% accurate Expert 3 is 85% accurate They are ...
0
votes
0answers
21 views

Test-Retest reliability/ICC Help?

I’m sure this is a simple question but I’m new to statistics and would really appreciate the help! The study I am working collected data over two testing days. On each testing day subjects ran a ...
0
votes
0answers
14 views

Measuring IRR between groups as well as improvement in agreement within groups

I have been going back and forth on proper stats for this question, and I am stumped. Basic design is a 2 x 3 mixed design. Two between-subject groups - both are measured for their opinions at ...
0
votes
0answers
8 views

Should I standardize questionnaire responses prior to reliability analysis?

I have a multiple questionnaires with responses on Likert scales. They differ in that some are rated on a 7p scale, and others on a 5p scale. I would like to look at the reliability of each scale as ...
1
vote
1answer
24 views

d to r conversion for extremely high versus extremely low groups

Context Suppose I am performing a meta-analysis on the relationship between two variables (say depression and life satisfaction). Unfortunately, many studies in the prior literature do not report ...
0
votes
0answers
21 views

Does reliability of measurement affect the minimal sample size required?

The following calculator (like many of its kind) uses a number of variables to determine an adequate (and importantly, representative) sample size: https://www.calculator.net/sample-size-calculator....
0
votes
0answers
13 views

Reliability resulting from Classification + Discriminant Analysis

I have the task to classify 90 small lakes into one of the classes "very salty", "moderately salty", "not salty". I use two different methods to do that. Method 1 is based on Electric Conductivity ...
4
votes
0answers
237 views

Intraclass correlation coefficient (ICC) for two raters using a mixed effects model for a design with repeated measurements

I want to calculate the intraclass correlation coefficient (ICC) for an inter-rater assessment with two raters. The design is as follows: Each of the $n$ subjects were assessed by the same 2 raters on ...
0
votes
0answers
47 views

Which ICC (ICC(2,1) or ICC(2,k)) should I report?

I have some hundred subjects indicating on a 7point scale for 10 situations how likely they are to be influenced in diverse situations. I will build an index of the responses for these behaviors and ...
0
votes
0answers
21 views

Tau, congeneric and parallel indicator

I'm trying to calculate with r a tau, congeneric and parallel model starting from a matrix correlation but I don't seem to manage to get it right. I'v particular trouble with the tau equivalent model. ...
0
votes
0answers
16 views

How to assess “reliability” for a non-stationary time series compared to a reference

I have two arterial pressure measurement devices, one of which is the reference. My goal is to assess reliability compared to the reference and the variables (demographics, passing time, etc.) ...
1
vote
0answers
43 views

Using the Standard Error of Prediction in the presence of practice effects

I’m wondering about the following hypothetical scenario. There’s a student who previously scored 40% on an examination with a pass mark of 50%, a mean mark of 60%, SD of 10 and a reliability of 0.6. ...
0
votes
0answers
14 views

How to Compute Krippendorff's Alpha for Text Continua?

I have a question about the inter-annotator reliability measure of Krippendorff's alpha for text continua or text unitization. The measure is explained in detail in the paper. I have texts and 2 ...
0
votes
1answer
66 views

How to calculate intra-observer error and average difference?

I am trying to assess intra-observer error in a setting where measurements where done repeatedly by a single observer. So far I'm having some conceptual and technical issues. In the dataset we have ...
0
votes
0answers
24 views

Krippendorff's Alpha for unitizing texts

I have a question about using alpha to measure Krippendorff's inter-annotator reliability in qualitative data. After reading the papers, I'm still not sure how to handle this case. I'm working on a ...
0
votes
1answer
25 views

Reliability Testing, Determining Weibull parameters

I want to use a Weibull analysis for demonstrating the reliability of a piece of equipment that will be set to cycle. How do I select the proper scale and shape parameters for the distribution if I ...
0
votes
0answers
24 views

first factor saturation vs. general factor saturation

a simple explanation between these two in the context of reliability analysis, specifically Cronbach's alpha.
1
vote
1answer
33 views

Unidimensionality in IRT

I recently came across the definition of unidimensionality (in the context of IRT and internal consistency reliability, specifically Cronbach's alpha). Johnsosn (2007) states that There is a one-...
0
votes
0answers
38 views

Cronbach's alpha interpretation - dichotomous data

I have calculated Cronbach's alpha in R using the psych package and I am trying to interpret them now. I want to properly understand this, as it is part of my ...
1
vote
1answer
37 views

Correlations - multiple observations & multiple observers

I haven't found quite this question on here. I have a data set that typically requires generalizability theory to untangle however I now face a question I believe g theory can not answer. 17 ...
0
votes
0answers
6 views

MTBF for a single failure

I have a machine for which I want to calculate the MTBF over a limited period of time. The failure rate is small and not constant. That is, I may encounter a single failure (or no failures) over a ...
3
votes
1answer
92 views

Prediction interval for a Weibull distribution

Suppose I am producing units and have tested the failure time of some proportion of them (say 10%), possibly with some right censoring if they didn't fail within the testing period. I'm able to fit a ...
0
votes
1answer
209 views

item-total correlation vs. inter-item correlation

I am assesing the reliability and validity of an 16-item multuiple choice test supposed to test language skills. Cronbach's apha is "questionable" with .65. I am having troubles interpreting the ...
1
vote
0answers
17 views

Measuring equivalence/reliability (ICC and consorts)

For a reliability study I’m administering patient reported outcomes (questionnaires) with 2 modes of administration (on paper and via an electronic device). I’d like to measure equivalence/reliability ...
0
votes
0answers
269 views

Difference between Cronbach's alpha and Pearson's coefficient

I am creating an indicator for social development which includes variables of health, education and economy. Since I have many variables, I decided to remove some of them based on dominion expertise ...
0
votes
0answers
10 views

Inter-rater reliability many coders but only one case

I have a specific performance assessment metric which I aim to get some reliability and validity data for. And I really would appreciate some help for the reliability: Background In my research, ...
1
vote
1answer
101 views

Item response theory for continuous variables, and estimating standard error of measurement

I really like how traditional item response theory (IRT) packages tell you the standard error of measurement conditional on one's ability level, and from that, you can calculate the test information ...
0
votes
0answers
44 views

How to calculate overall reliability for an IRT test with subtests

I have an IRT, 2 parameter model ability test with three subtests - verbal, numerical and abstract reasoning. I have been able to calculate the reliability of each subtest using the methods outlined ...
2
votes
1answer
48 views

Proof of direct proportionality between hazard rate function and probability density function

I'm reading up on reliability and I came across this question: Show that if the hazard function is decreasing, the PDF, $f(t)$, is also a decreasing function and its mode must therefore occur at t =...
0
votes
0answers
51 views

Generating Failure / Suspension data from Weibull Distribution

I've been trying to make some synthetic failure data in MATLAB and I'm coming across an issue when I try to refit the Weibull distribution from my synthetic data. For example, assuming a Weibull ...