Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Cohens Kappa for multiple categories

I have a data set of two observers evaluating 50 patient xrays. We are evaluating hip replacement femoral stem loosening by gruen zone, meaning a single patient may have more than one zone of ...
1
vote
0answers
8 views

Denotion of Cronbach's Alpha based on Polychorric Correlations

we assessed several 5 point Likert-Scale items in a questionnaire and are interested in Cronbach's $\alpha$ for the internal consistency. I recently read (here) that it can be more precise to used ...
0
votes
1answer
14 views

Bernoulli Trials with Bayesian Analysis

Can Bayesian analysis be used with multi-class problems? For example, reliability analysis is usually concerned with "failed" or "not failed" classifications. My understanding is ...
1
vote
0answers
32 views

Interpretation of Intraclass correlation coefficient (ICC) as a reliability estimate for hierarchical designs - lme4

I am working on the following problem and would really appreciate your help: The data set at hand contains medical ratings (scale 1-5, assumed interval) by 5 independent raters, on 7 occasions, for 60 ...
0
votes
0answers
9 views

When conducting a reliability test using Cronbach's alpha, is it better to conduct the reliability test on the raw data or the cleaned data?

I'm working on a longitudinal study, and I had to take out some participants who didn't meet the required number of days completed. When conducting a reliability test using Cronbach's alpha, is it ...
2
votes
0answers
28 views

Is there any way to visualize Cronbach's alpha value and test-retest reliability (intraclass correlation coefficient) analysis outcomes?

Is there any way to visualize Cronbach's alpha value and test-retest reliability (intraclass correlation coefficient) analysis outcome? I use Rstudio as my daily driver, so if there's any way to ...
1
vote
0answers
19 views

Simulating an example of correlation attenuation due to measurement error

I'm trying to simulate an example of the attenuation of an association due to measurement error. According to the attenuation formula from classical test theory, the presence of measurement error ...
0
votes
0answers
9 views

power, correlations and difference scores

Would it make a difference for a statistical power calculation whether the correlations are obtained using the simple measures or whether they are difference scores? Would you need more participants ...
0
votes
1answer
49 views

Application of Metropolis Hastings

I am trying to implement the Metropolis Hastings algorithm for Bayesian analysis. In this case, the parameter of interest is the scale parameter for a Weibull distribution. The context is for ...
2
votes
1answer
30 views

The correction for attenuation

The reliability of the task that I measured is very low for some sub-tasks. So, I tried to correct the correlation coefficient with Spearman's disattenuation formula: $r_{x'y'} = r_{xy}/\sqrt(r_{xx}r_{...
1
vote
0answers
96 views

Interpreting Omega coefficients (reliability) from EFA and CFA

I have a dataset with 5 variables and I would like to estimate the reliability of the scale score computed as the linear combination of all items (n = number of items), that is $O = \sum_{i=1}^{n}(X_i)...
0
votes
0answers
51 views

Best suitable type of Intraclass correlation method?

I would like to understand if the below data is suitable to run single_raters_absolute IntraClass Correlation (ICC) or Single_random_raters or Single_fixed_raters. I have executed the below program in ...
3
votes
1answer
110 views

How to do exploratory factor analysis in a repeated measure design?

In my experiment participants answered to the same scale twice, after the presentation of two different stimuli, i.e., repeated measures design. The scale validation is not the main focus of my study, ...
1
vote
0answers
31 views

Linear mixed model for analysis between 2 methods and intra inter observer reliability/agreement

i have ultrasound data measuring strain of myocardium as the dependant variable. Now i want to compare reliability and agreement of the measurement system between the 2 methods, the test retest and ...
1
vote
0answers
39 views

How to get the “reliability percentage” of a measurement

I need to get the amount of confidence I can have on a measurement (from 0 to 100 percent). In other words, I need to know how much reliable the data on a measurement is for me to trust its ...
1
vote
1answer
19 views

How to define covariates for reliability - Cox regression

Hope you are all doing great and staying safe during this pandemic. Here is my question about defining covariates for Cox regression. There are structures where I would like to model their reliability ...
1
vote
0answers
15 views

Cronbachs alpha for items with different scales

I have a sample size of N=63 and 6 items measuring risk tolerance. Now, I want to calculate Cronbachs alpha, however, the 6 items have different scales. For example, the first item has a scale from 1-...
0
votes
0answers
6 views

Does low test-retest reliability lead to low replicability?

I have read the following paper: https://journals.sagepub.com/doi/10.1177/0146167211400619 which shows that poor reliability may lead to reproducibility issues. However, there was a paper in response ...
0
votes
0answers
17 views

Evaluating discriminant validity of a 2nd order reflective construct?

I am currently assessing my measurement model which includes-besides other constructs-a 2nd order reflective construct (consists of four 1st order constructs that are also reflective). I noticed that ...
0
votes
2answers
27 views

Is this an instance of convergent evidence?

In psychometrics, convergent validity is a source of proof for construct validity. It is said that it is the degree to which test scores are correlated with test scores of relevant constructs or the ...
1
vote
1answer
40 views

Can a binary output model with auc 0.5 be perfectly calibrated?

I am reading up on model calibration and I stumbled upon this article. To quote: We can have a perfectly accurate model that is not calibrated at all and, on the other hand, a model that is no better ...
0
votes
1answer
18 views

When running a reliability analysis (using Cronbach's alpha) in SPSS, what does “scale” refer to? Do I have to compute a scale explicitly beforehand?

Is the scale meant to represent the sum of responses (each response is an element of, say, the set {1, 2, 3, 4, 5}) of a case (an individual) to a set of questions (items)? And is a "scale mean&...
0
votes
0answers
53 views

Test-Retest reliability with multiple raters on different subjects at different times

I want to be able to calculate the agreement between two ratings measured at two separate times. The rating is on a scale of 1-5. For example, the below data relates to 3 different "Rater"'s ...
0
votes
0answers
7 views

Reliability influence of teststructure to shape of distribution

following problem: We have a chain of a known size (geometry of the chain links) and number of chain links. The chain is on a horizontal table to avoid additional mechanical load due to influence of ...
0
votes
0answers
12 views

Adaptive Sampling techniques for SVM?

I am an Engineer interested in creating a surrogate model of a certain phenomenon in the context of reliability engineering. Essentially my quantity of interest is the Limit state function/stability ...
0
votes
0answers
18 views

Spearman's Correction for Attenuation - What type of reliability?

I am trying to correct a correlation coefficient with Spearman's disattenuation formula: rx'y' = rxy/sqrt(rxx*ryy) The equation is pretty straightforward, however, the literature always talks about ...
6
votes
2answers
135 views

Conditional intraclass correlation (ICC) from a linear mixed model as evidence for test-retest reliability?

In my experiment with two conditions (between-subjects design), participants completed a single-item scale three times: (1) before the experimental manipulation, (2) after the experimental ...
0
votes
0answers
40 views

Python Random Forest Prediction Probabilities Reliability, Overfitting?

TLDR: RF prediction probabilities are not consistent I have created a calibrated Random Forest Model to predict probabilities for attrition of the workforce, but what I am finding is that ...
1
vote
1answer
46 views

Estimation of measurement reliability

I have the following statistical setting: I have two unknowns $a$ and $b$ an I have two measurements of their values corrupted by noise distibuted uniformly in $[-e,e]$ and I have exact value of $e$ ...
0
votes
1answer
35 views

Compare agreement between multiple different methods

There are 3 subjective methods to measure the severity of a condition. Each method measures it on an ordinal scale, but the scales are different: method 1 scale is 1-3 method 2 scale is 1-4 method 3 ...
0
votes
0answers
15 views

Sample size cutoff for agreement coefficients

New poster here. Would love some help thinking through this. Suppose I am observing individuals in public space, and I am recording the number of occurrences of a target behavior (e.g., looking at a ...
0
votes
0answers
49 views

Interrater reliability for several variables

I need to check the inter-observer reliability of a data i.e. how consistent the clinical assessment is when performed by two independent researchers. The data would look like this: ...
0
votes
0answers
9 views

How to interpret rising Cronbach's Alpha in later cohorts

I did conduct a study with test persons in three cohorts at the beginning (A), middle (B) and towards the end (C) of their education. I used several likert scales which I now want to check for ...
1
vote
0answers
20 views

Determining whether a rating scale is ordinal or nominal from data?

I am occasionally presented with data where a rating scale was used (i.e. options 1 through 5). Sometimes these are ordinal, e.g. Very positive. Somewhat positive Neutral Somewhat negative Very ...
0
votes
0answers
19 views

Comparing groups, distinctly fitted with Weibull models, via the likelihood ratio test

I'm working with some data on the fatigue (i.e. time to failure under cyclic loads) of a certain alloy. I have four groups of data, each with the 'treatment' of a slightly different manufacturing ...
0
votes
0answers
22 views

Std error of measurement and CIs for inter-rater observations

Reliability is often reported somewhat abstractly and can be difficult to interpret (e.g., ICC = 0.9). For test-retest reliability, it can be helpful to express measurement error in the scale of the ...
1
vote
0answers
25 views

How to compare inter-rater reliability between two time points?

We have two sets of raters judge the same items before and after a training, resulting in an ICC and kappa value for each time point (pre- and post-training). How would you statistically compare pre- ...
0
votes
0answers
27 views

How do I calculate the parameters of the beta distribution from the mean and the variance? [duplicate]

My mean is 10 and the variance is 4. There has been confusion around the bounds of $\mu$ and $\sigma_2$ for any given Beta distribution, what I understand is, $$ \mu = \dfrac{\alpha}{\alpha + \beta} ...
3
votes
0answers
70 views

Does it make sense to use the Spearman-Brown correction (prediction / prophecy) formula with Kendall's Tau correlation coefficient?

The Spearman-Brown prophecy formula is used to estimate the increase in reliability that is obtained when more measurements are taken (and aggregated) from the same construct. Reliability is the ...
1
vote
0answers
48 views

How to use Cronbach's Alpha?

Is it appropriate to use Cronbach's Alpha for nominal variables? I have 20 nominal variables (two dimensions with 10 questions each) that ask about the characteristics of work performance. The ...
0
votes
0answers
47 views

Maximum likelihood estimate of θ

Here's the question im trying to solve. Correct me if im wrong please. S(360) means the deaths that occured in the time period of 0 to 360 hours. Does S7 means that there will be seven deaths during ...
0
votes
0answers
78 views

sample size calculation for intraclass correlation coefficient (ICC)

I want to design a study when I'll be comparing three measurements of a variable in imaging and then compute an ICC as a measure of agreement since I'll be obtain a continuous measurement. I read ...
0
votes
0answers
15 views

System Yield in Reliability

Let $Y(t)$ be the system yield function: \begin{equation} Y(t) = \int_0^\infty [v(t) - c(t)]dt, \end{equation} where $v(t)$ is the revenue at time $t$ and $c(t)$ is the cost at time $t$. The ...
0
votes
0answers
25 views

Is bootstrapping the correct way to prove reliability/unreliability of a statistic?

I have a sample of which I calculated an average, say m (the sample size was calculated using a confidence of 0.95 and error of 0.02; see RaoSoft calculator. So the population average should fall ...
0
votes
1answer
12 views

Can reliability and validity assessments be considered as required constituent of the “Measurement part” in Structural Equation Modeling?

I wonder if any type of reliability and / or validity analysis can be considered as an element of the Measurement part in SEM. For example, when computing Cronbach's alpha or the AVEs of an instrument,...
2
votes
0answers
175 views

Platt Scaling vs Isotonic Regression

I am learning classifier probability calibrations and have calibrated an eleastic net model using both Platt scaling and isotonic regression. As you can see in the attached image Platt scaling (on the ...
0
votes
0answers
10 views

Does the quoted line mean that the 6 cars would have gained more miles? If not, what does it mean?

I am reading Jayant Deshpande's "Life Time Data: Statistical Models and Methods" book and while reading about Right Random Censoring, I read about this example. Example of Random (right) Censoring ...
1
vote
0answers
9 views

Measuring Validity and reliability

Say you have an independent train ticket system where automatic refunds would be issued in case of a delayed arrival. To determine the arrival time of the train at a certain station you collect data ...
1
vote
1answer
111 views

Reliability in R

I'm having trouble understanding the difference between omega 1 and omega 3 in reliability function in the semTools package in <...
0
votes
0answers
8 views

Custom Zero-Fail Data Analysis Method

I'm trying to analyze products with zero-failure and get reliability values. I've looked into other methods here. Some of my products have very few observations and some have several hundred. I've ...

1
2 3 4 5
9