Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

Filter by
Sorted by
Tagged with
2
votes
1answer
47 views

How to do exploratory factor analysis in a repeated measure design?

In my experiment participants answered to the same scale twice, after the presentation of two different stimuli, i.e., repeated measures design. The scale validation is not the main focus of my study, ...
1
vote
0answers
26 views

Linear mixed model for analysis between 2 methods and intra inter observer reliability/agreement

i have ultrasound data measuring strain of myocardium as the dependant variable. Now i want to compare reliability and agreement of the measurement system between the 2 methods, the test retest and ...
1
vote
0answers
34 views

How to get the “reliability percentage” of a measurement

I need to get the amount of confidence I can have on a measurement (from 0 to 100 percent). In other words, I need to know how much reliable the data on a measurement is for me to trust its ...
1
vote
1answer
16 views

How to define covariates for reliability - Cox regression

Hope you are all doing great and staying safe during this pandemic. Here is my question about defining covariates for Cox regression. There are structures where I would like to model their reliability ...
1
vote
0answers
12 views

Cronbachs alpha for items with different scales

I have a sample size of N=63 and 6 items measuring risk tolerance. Now, I want to calculate Cronbachs alpha, however, the 6 items have different scales. For example, the first item has a scale from 1-...
0
votes
0answers
5 views

Does low test-retest reliability lead to low replicability?

I have read the following paper: https://journals.sagepub.com/doi/10.1177/0146167211400619 which shows that poor reliability may lead to reproducibility issues. However, there was a paper in response ...
0
votes
0answers
8 views

Evaluating discriminant validity of a 2nd order reflective construct?

I am currently assessing my measurement model which includes-besides other constructs-a 2nd order reflective construct (consists of four 1st order constructs that are also reflective). I noticed that ...
0
votes
2answers
21 views

Is this an instance of convergent evidence?

In psychometrics, convergent validity is a source of proof for construct validity. It is said that it is the degree to which test scores are correlated with test scores of relevant constructs or the ...
1
vote
1answer
31 views

Can a binary output model with auc 0.5 be perfectly calibrated?

I am reading up on model calibration and I stumbled upon this article. To quote: We can have a perfectly accurate model that is not calibrated at all and, on the other hand, a model that is no better ...
0
votes
1answer
15 views

When running a reliability analysis (using Cronbach's alpha) in SPSS, what does “scale” refer to? Do I have to compute a scale explicitly beforehand?

Is the scale meant to represent the sum of responses (each response is an element of, say, the set {1, 2, 3, 4, 5}) of a case (an individual) to a set of questions (items)? And is a "scale mean&...
0
votes
0answers
20 views

Test-Retest reliability with multiple raters on different subjects at different times

I want to be able to calculate the agreement between two ratings measured at two separate times. The rating is on a scale of 1-5. For example, the below data relates to 3 different "Rater"'s ...
0
votes
0answers
7 views

Reliability influence of teststructure to shape of distribution

following problem: We have a chain of a known size (geometry of the chain links) and number of chain links. The chain is on a horizontal table to avoid additional mechanical load due to influence of ...
0
votes
0answers
10 views

Adaptive Sampling techniques for SVM?

I am an Engineer interested in creating a surrogate model of a certain phenomenon in the context of reliability engineering. Essentially my quantity of interest is the Limit state function/stability ...
0
votes
0answers
15 views

Spearman's Correction for Attenuation - What type of reliability?

I am trying to correct a correlation coefficient with Spearman's disattenuation formula: rx'y' = rxy/sqrt(rxx*ryy) The equation is pretty straightforward, however, the literature always talks about ...
6
votes
1answer
69 views

Conditional intraclass correlation (ICC) from a linear mixed model as evidence for test-retest reliability?

In my experiment with two conditions (between-subjects design), participants completed a single-item scale three times: (1) before the experimental manipulation, (2) after the experimental ...
0
votes
0answers
35 views

Python Random Forest Prediction Probabilities Reliability, Overfitting?

TLDR: RF prediction probabilities are not consistent I have created a calibrated Random Forest Model to predict probabilities for attrition of the workforce, but what I am finding is that ...
1
vote
1answer
41 views

Estimation of measurement reliability

I have the following statistical setting: I have two unknowns $a$ and $b$ an I have two measurements of their values corrupted by noise distibuted uniformly in $[-e,e]$ and I have exact value of $e$ ...
0
votes
1answer
34 views

Compare agreement between multiple different methods

There are 3 subjective methods to measure the severity of a condition. Each method measures it on an ordinal scale, but the scales are different: method 1 scale is 1-3 method 2 scale is 1-4 method 3 ...
0
votes
0answers
15 views

Sample size cutoff for agreement coefficients

New poster here. Would love some help thinking through this. Suppose I am observing individuals in public space, and I am recording the number of occurrences of a target behavior (e.g., looking at a ...
0
votes
0answers
29 views

Interrater reliability for several variables

I need to check the inter-observer reliability of a data i.e. how consistent the clinical assessment is when performed by two independent researchers. The data would look like this: ...
0
votes
0answers
9 views

How to interpret rising Cronbach's Alpha in later cohorts

I did conduct a study with test persons in three cohorts at the beginning (A), middle (B) and towards the end (C) of their education. I used several likert scales which I now want to check for ...
0
votes
0answers
13 views

Determining whether a rating scale is ordinal or nominal from data?

I am occasionally presented with data where a rating scale was used (i.e. options 1 through 5). Sometimes these are ordinal, e.g. Very positive. Somewhat positive Neutral Somewhat negative Very ...
0
votes
0answers
9 views

Comparing groups, distinctly fitted with Weibull models, via the likelihood ratio test

I'm working with some data on the fatigue (i.e. time to failure under cyclic loads) of a certain alloy. I have four groups of data, each with the 'treatment' of a slightly different manufacturing ...
0
votes
0answers
18 views

Std error of measurement and CIs for inter-rater observations

Reliability is often reported somewhat abstractly and can be difficult to interpret (e.g., ICC = 0.9). For test-retest reliability, it can be helpful to express measurement error in the scale of the ...
0
votes
0answers
12 views

How to compare inter-rater reliability between two time points?

We have two sets of raters judge the same items before and after a training, resulting in an ICC and kappa value for each time point (pre- and post-training). How would you statistically compare pre- ...
0
votes
0answers
23 views

How do I calculate the parameters of the beta distribution from the mean and the variance? [duplicate]

My mean is 10 and the variance is 4. There has been confusion around the bounds of $\mu$ and $\sigma_2$ for any given Beta distribution, what I understand is, $$ \mu = \dfrac{\alpha}{\alpha + \beta} ...
1
vote
0answers
31 views

Does it make sense to use the Spearman-Brown correction (prediction / prophecy) formula with Kendall's Tau correlation coefficient?

The Spearman-Brown prophecy formula is used to estimate the increase in reliability that is obtained when more measurements are taken (and aggregated) from the same construct. Reliability is the ...
1
vote
0answers
32 views

How to use Cronbach's Alpha?

Is it appropriate to use Cronbach's Alpha for nominal variables? I have 20 nominal variables (two dimensions with 10 questions each) that ask about the characteristics of work performance. The ...
0
votes
0answers
47 views

Maximum likelihood estimate of θ

Here's the question im trying to solve. Correct me if im wrong please. S(360) means the deaths that occured in the time period of 0 to 360 hours. Does S7 means that there will be seven deaths during ...
0
votes
0answers
27 views

sample size calculation for intraclass correlation coefficient (ICC)

I want to design a study when I'll be comparing three measurements of a variable in imaging and then compute an ICC as a measure of agreement since I'll be obtain a continuous measurement. I read ...
0
votes
0answers
13 views

System Yield in Reliability

Let $Y(t)$ be the system yield function: \begin{equation} Y(t) = \int_0^\infty [v(t) - c(t)]dt, \end{equation} where $v(t)$ is the revenue at time $t$ and $c(t)$ is the cost at time $t$. The ...
0
votes
0answers
25 views

Is bootstrapping the correct way to prove reliability/unreliability of a statistic?

I have a sample of which I calculated an average, say m (the sample size was calculated using a confidence of 0.95 and error of 0.02; see RaoSoft calculator. So the population average should fall ...
0
votes
0answers
10 views

Can reliability and validity assessments be considered as required constituent of the “Measurement part” in Structural Equation Modeling?

I wonder if any type of reliability and / or validity analysis can be considered as an element of the Measurement part in SEM. For example, when computing Cronbach's alpha or the AVEs of an instrument,...
2
votes
0answers
101 views

Platt Scaling vs Isotonic Regression

I am learning classifier probability calibrations and have calibrated an eleastic net model using both Platt scaling and isotonic regression. As you can see in the attached image Platt scaling (on the ...
0
votes
0answers
10 views

Does the quoted line mean that the 6 cars would have gained more miles? If not, what does it mean?

I am reading Jayant Deshpande's "Life Time Data: Statistical Models and Methods" book and while reading about Right Random Censoring, I read about this example. Example of Random (right) Censoring ...
1
vote
0answers
9 views

Measuring Validity and reliability

Say you have an independent train ticket system where automatic refunds would be issued in case of a delayed arrival. To determine the arrival time of the train at a certain station you collect data ...
1
vote
1answer
71 views

Reliability in R

I'm having trouble understanding the difference between omega 1 and omega 3 in reliability function in the semTools package in <...
0
votes
0answers
8 views

Custom Zero-Fail Data Analysis Method

I'm trying to analyze products with zero-failure and get reliability values. I've looked into other methods here. Some of my products have very few observations and some have several hundred. I've ...
1
vote
0answers
28 views

Emergency Use Auth (EUA) for SARS-CoV-2 Tests

Problem: After the EUA for SARS-CoV-2 tests, we have a diagnostic test that passes both of the following criteria: All first five true-negative samples each produce a negative test result. All ...
1
vote
0answers
33 views

How to test the disturbance effect of a measuring instrument?

I am trying to think of an experimental design that allows me to test to what extent a measuring tool (automatic counting unit) alters (by restricting the entrance area) the measured quantity (number ...
0
votes
0answers
11 views

Difference between inter-itemcorrelation and Cronbach's alpha

What exactly is the difference between inter-itemcorrelation and Cronbach's alpha? Is the one better than the other when there is, for example, a lower amount of items (e.g., 4 items)? Thank you!
0
votes
0answers
21 views

Bland-Altman analysis for repeated measure across time with one method of measurement

Just like to seek some clarification. As far as I understand and have read "The Bland–Altman method calculates the mean difference between two methods of measurement (the ‘bias’), and 95% limits of ...
0
votes
0answers
30 views

Gamma to Beta: Successes and Failure Proportions?

I have a question after reading this article on the relation between Gamma and Beta distributions and this article on the intuitive understanding of the Beta Distribution. Question As the Gamma ...
2
votes
1answer
107 views

Inter rater reliability with multiple raters, multiple categorical variables with different levels

Apologies if the answer to this question exists elsewhere, but I haven't found anything quite matching what I'm looking for. I have a dataset wherein 3 raters independently assessed 10 subjects by ...
0
votes
0answers
74 views

AVE and CR in JASP?

I'm new to the software JASP and need some help. Is it possible to calculate average variance extracted (AVE) and composite reliability (CR) in JASP? I need them in order to test validity and ...
0
votes
0answers
22 views

Internal consistency: Alternative measures for unidimensional scales with a small number of items?

I use r and analyze a scale of 11 items, theoretically assuming one global factor and three group factors. Since a CFA revealed numerous correlated errors and due to the criticism of cronbachs alpha (...
2
votes
0answers
42 views

comparison between two measurement devices

I have two measurement devices accelerometer A and B. The measurement is the step time of a person walking in a corridor. A is the gold standard. I want to check the validity and reliability of B. how ...
1
vote
2answers
26 views

Reliability of a multistate item

I have a question and appreciate your time and input to give me some hints. If a system could stay in several states ranked based on the performance of the system such as (state 8 - perfect state, ...
0
votes
1answer
33 views

Inferential interpretation of Kappa reliability

I have obtained a Fliess' Kappa interrater reliability of $37.5\%$ from $90$ randomly selected raters rating $100$ randomly selected, binary items ($Yes/No$). Question: Is the following ...
1
vote
0answers
17 views

Can I use G theory for this case? How? Comparing Waldorf and traditional schools

I have my data in the format of: In my study I examined 20 waldorf student and 30 traditional student using concept map. Concept maps can be scored by different scoring methods. How can I determine ...

1
2 3 4 5
9