Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Analyzing repeatability using rptR package with binary data, CI doesn't match estimated R [on hold]

I'm using the rptR package in R with the following code: rpt(SUCCESS ~ TRIAL + (1|SUBJECT), data=phase1, datatype ="Binary", grname="SUBJECT", nboot = 100, npermut = 0, ratio = TRUE) I've also ran ...
0
votes
0answers
14 views

TTF and Relibility data form

Assume the car engine has 4 failure in one month. Example: Failure at day 10, 15, 23, 29. Is TTF maintained after each failure as shown in the following table, or it is increasing? How to calculate ...
0
votes
1answer
15 views

Which distribution or process should be used for wearout reliability modeling?

When modeling the reliability of a system, it is usual to use exponential distribution to model errors that occur randomly throughout the system's useful lifetime (the middle part of the well-known ...
0
votes
0answers
7 views

inter-observer reliability

I want to test the inter observer reliability between myself and one other observer on three different groups of data. I am unsure hat the best statistic is to go with the data I have collected: 1) ...
0
votes
0answers
21 views

Reliability estimation for questionnaire - is Cronbach's a applicable?

I am fairly new to scale analysis and I need to estimate the reliability and validity of a 5 item scale (5 questions, 2 have 4 choices abcd and 3 have 2 choices ab). The scale is supposed to be ...
0
votes
0answers
29 views

Minitab / Survival Analysis / Multiply Right Censor

I am doing an analysis to see if investments in leadership training have decreased attrition. The data I have is: Employee Start Date Hire Quarter Term Date (if not still employed) Status (Active ...
0
votes
0answers
19 views

Intra-class Correlation Coefficient (ICC) for a single subject

I need to calculate a score that reflects the level of disagreement/ambiguity between raters for a music clip. The experimental set-up is that 10 respondents listen to a clip with a one-dimensional ...
0
votes
0answers
13 views

Cohen's Kappa (observer variability) with partially ordered sets

I was wondering if there is a common way to compute Cohen's k when the values are partially ordered sets (of a lattice). My data roughly looks like the table below. There is a default lower bound (...
5
votes
0answers
44 views

Bootstrap Prediction Interval: which residuals to use and which method?

I ask this question referring to the post: Bootstrap prediction interval, where a step by step method for calculating the prediction interval for linear regression models is explained. In the ...
0
votes
1answer
35 views

Real-life application of min(X,Y) series system

Let $X$ and $Y$ be the random variables. What are the real life situations where $\min(X,Y)$ arises?
1
vote
1answer
44 views

Computing the accuracy of an answer

Let's assume I have three experts providing an answer. Expert 1 is 95% accurate (The likelihood of providing a correct answer is 95%). Expert 2 is 90% accurate Expert 3 is 85% accurate They are ...
0
votes
0answers
15 views

Test-Retest reliability/ICC Help?

I’m sure this is a simple question but I’m new to statistics and would really appreciate the help! The study I am working collected data over two testing days. On each testing day subjects ran a ...
0
votes
0answers
13 views

Measuring IRR between groups as well as improvement in agreement within groups

I have been going back and forth on proper stats for this question, and I am stumped. Basic design is a 2 x 3 mixed design. Two between-subject groups - both are measured for their opinions at ...
0
votes
0answers
8 views

Should I standardize questionnaire responses prior to reliability analysis?

I have a multiple questionnaires with responses on Likert scales. They differ in that some are rated on a 7p scale, and others on a 5p scale. I would like to look at the reliability of each scale as ...
1
vote
1answer
23 views

d to r conversion for extremely high versus extremely low groups

Context Suppose I am performing a meta-analysis on the relationship between two variables (say depression and life satisfaction). Unfortunately, many studies in the prior literature do not report ...
0
votes
0answers
20 views

Does reliability of measurement affect the minimal sample size required?

The following calculator (like many of its kind) uses a number of variables to determine an adequate (and importantly, representative) sample size: https://www.calculator.net/sample-size-calculator....
0
votes
0answers
13 views

Reliability resulting from Classification + Discriminant Analysis

I have the task to classify 90 small lakes into one of the classes "very salty", "moderately salty", "not salty". I use two different methods to do that. Method 1 is based on Electric Conductivity ...
4
votes
0answers
178 views

Intraclass correlation coefficient (ICC) for two raters using a mixed effects model for a design with repeated measurements

I want to calculate the intraclass correlation coefficient (ICC) for an inter-rater assessment with two raters. The design is as follows: Each of the $n$ subjects were assessed by the same 2 raters on ...
0
votes
0answers
36 views

Which ICC (ICC(2,1) or ICC(2,k)) should I report?

I have some hundred subjects indicating on a 7point scale for 10 situations how likely they are to be influenced in diverse situations. I will build an index of the responses for these behaviors and ...
0
votes
0answers
14 views

Tau, congeneric and parallel indicator

I'm trying to calculate with r a tau, congeneric and parallel model starting from a matrix correlation but I don't seem to manage to get it right. I'v particular trouble with the tau equivalent model. ...
0
votes
0answers
14 views

How to assess “reliability” for a non-stationary time series compared to a reference

I have two arterial pressure measurement devices, one of which is the reference. My goal is to assess reliability compared to the reference and the variables (demographics, passing time, etc.) ...
0
votes
0answers
35 views

Using the Standard Error of Prediction in the presence of practice effects

I’m wondering about the following hypothetical scenario. There’s a student who previously scored 40% on an examination with a pass mark of 50%, a mean mark of 60%, SD of 10 and a reliability of 0.6. ...
0
votes
0answers
13 views

How to Compute Krippendorff's Alpha for Text Continua?

I have a question about the inter-annotator reliability measure of Krippendorff's alpha for text continua or text unitization. The measure is explained in detail in the paper. I have texts and 2 ...
0
votes
1answer
32 views

How to calculate intra-observer error and average difference?

I am trying to assess intra-observer error in a setting where measurements where done repeatedly by a single observer. So far I'm having some conceptual and technical issues. In the dataset we have ...
0
votes
0answers
20 views

Krippendorff's Alpha for unitizing texts

I have a question about using alpha to measure Krippendorff's inter-annotator reliability in qualitative data. After reading the papers, I'm still not sure how to handle this case. I'm working on a ...
0
votes
1answer
23 views

Reliability Testing, Determining Weibull parameters

I want to use a Weibull analysis for demonstrating the reliability of a piece of equipment that will be set to cycle. How do I select the proper scale and shape parameters for the distribution if I ...
0
votes
0answers
14 views

first factor saturation vs. general factor saturation

a simple explanation between these two in the context of reliability analysis, specifically Cronbach's alpha.
1
vote
1answer
31 views

Unidimensionality in IRT

I recently came across the definition of unidimensionality (in the context of IRT and internal consistency reliability, specifically Cronbach's alpha). Johnsosn (2007) states that There is a one-...
0
votes
0answers
28 views

Cronbach's alpha interpretation - dichotomous data

I have calculated Cronbach's alpha in R using the psych package and I am trying to interpret them now. I want to properly understand this, as it is part of my ...
1
vote
1answer
34 views

Correlations - multiple observations & multiple observers

I haven't found quite this question on here. I have a data set that typically requires generalizability theory to untangle however I now face a question I believe g theory can not answer. 17 ...
0
votes
0answers
6 views

MTBF for a single failure

I have a machine for which I want to calculate the MTBF over a limited period of time. The failure rate is small and not constant. That is, I may encounter a single failure (or no failures) over a ...
3
votes
1answer
71 views

Prediction interval for a Weibull distribution

Suppose I am producing units and have tested the failure time of some proportion of them (say 10%), possibly with some right censoring if they didn't fail within the testing period. I'm able to fit a ...
0
votes
1answer
96 views

item-total correlation vs. inter-item correlation

I am assesing the reliability and validity of an 16-item multuiple choice test supposed to test language skills. Cronbach's apha is "questionable" with .65. I am having troubles interpreting the ...
1
vote
0answers
15 views

Measuring equivalence/reliability (ICC and consorts)

For a reliability study I’m administering patient reported outcomes (questionnaires) with 2 modes of administration (on paper and via an electronic device). I’d like to measure equivalence/reliability ...
0
votes
0answers
177 views

Difference between Cronbach's alpha and Pearson's coefficient

I am creating an indicator for social development which includes variables of health, education and economy. Since I have many variables, I decided to remove some of them based on dominion expertise ...
0
votes
0answers
10 views

Inter-rater reliability many coders but only one case

I have a specific performance assessment metric which I aim to get some reliability and validity data for. And I really would appreciate some help for the reliability: Background In my research, ...
1
vote
1answer
81 views

Item response theory for continuous variables, and estimating standard error of measurement

I really like how traditional item response theory (IRT) packages tell you the standard error of measurement conditional on one's ability level, and from that, you can calculate the test information ...
0
votes
0answers
38 views

How to calculate overall reliability for an IRT test with subtests

I have an IRT, 2 parameter model ability test with three subtests - verbal, numerical and abstract reasoning. I have been able to calculate the reliability of each subtest using the methods outlined ...
2
votes
1answer
36 views

Proof of direct proportionality between hazard rate function and probability density function

I'm reading up on reliability and I came across this question: Show that if the hazard function is decreasing, the PDF, $f(t)$, is also a decreasing function and its mode must therefore occur at t =...
0
votes
0answers
34 views

Generating Failure / Suspension data from Weibull Distribution

I've been trying to make some synthetic failure data in MATLAB and I'm coming across an issue when I try to refit the Weibull distribution from my synthetic data. For example, assuming a Weibull ...
0
votes
1answer
19 views

Reliability analysis for different measurement scales?

I'm new here :) I'm conducting a survey questionnaire which gathers feedback on English-as Second Language (ESL). I have two measurement scales in my survey. Questions on how much time they spend ...
0
votes
0answers
38 views

Coefficient omega: appropriate extraction of factors

I use omega_hiearchical to determine the internal consistency of a scale. If I understood it correctly, it tries to extract three lower level factors by default. But you can specify that you want to ...
1
vote
0answers
15 views

Model building, how to include information on data point quality?

I have a dataset with count data predictors and a marker on their reliability or completeness. With 2 options: 0 - Data is not confirmed (could be higher); 1 - data is confirmed to be complete. The ...
0
votes
0answers
9 views

Best way to measure reliability of auto-correlated data

I'm trying to compare different ambulatory devices for measuring heart rate against a gold standard in-lab assessment. What is the best way to determine whether a device is worth using. I can use ...
0
votes
0answers
17 views

How much time is needed between measures for test-retest reliability?

I am planning a meta-analysis on reliability of various loneliness scales. We want to determine beforehand what time frame between measures we will consider. For instance, if the time between 2 ...
1
vote
1answer
164 views

Connection between Hazard Ratios, Survival, and Probability

According to this wiki link, the hazard ratio relates to survival according to the following equation $$ S_1(t)=S_0(t)^r \quad (1) $$ where $S$ is the survival and $r$ is the hazard ratio. So from ...
0
votes
0answers
17 views

Evidence fusion-Dempster Shafer

I have developed four models that estimate the same state-traffic volume in 100 streets during a specific time period. I have actual volume for 25 streets to estimate compute the reliability of each ...
0
votes
0answers
33 views

Can VAR models capture all frequency interactions reliably and to the same extent?

I would like to assess how a set of ~50-100 time-series variables causally interact with each other. To do this, I am fitting a VAR model and using cross-validation to estimate model-order. After the ...
0
votes
1answer
37 views

In scale adaptation which should be done first? CFA or Reliability Analysis

I am adapting questionnaire cross-cultural. Questinonnaire has 11 item and two factor. I want to confirm the structure 1 item's corrected item total correlation is <.20. Should this item be ...
0
votes
0answers
17 views

Measurement theory and sample size calculation for multivariate testing

Let $\mathbf{Y}$ be a vector of independent, normally-distributed random variables. Let $S_1$, $S_2$ and $S_3$ be three non-overlapping samples of sizes $N_1$, $N_2$ and $N_3$, respectively. Let $M_A$ ...