Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

Filter by
Sorted by
Tagged with
1
vote
0answers
12 views

How to process the test-retest reliability data of a questionnaire with essay questions?

I did a test-retest reliability on my survey. In my questionnaire, there are essay questions and the results are in the form of nominal data (Ex. : what is your favorite food?). I give a value of 1 if ...
0
votes
0answers
9 views

How to compare inter-rater reliability between two time points?

We have two sets of raters judge the same items before and after a training, resulting in an ICC and kappa value for each time point (pre- and post-training). How would you statistically compare pre- ...
0
votes
0answers
17 views

How do I calculate the parameters of the beta distribution from the mean and the variance? [duplicate]

My mean is 10 and the variance is 4. There has been confusion around the bounds of $\mu$ and $\sigma_2$ for any given Beta distribution, what I understand is, $$ \mu = \dfrac{\alpha}{\alpha + \beta} ...
1
vote
0answers
16 views

Does it make sense to use the Spearman-Brown correction (prediction / prophecy) formula with Kendall's Tau correlation coefficient?

The Spearman-Brown prophecy formula is used to estimate the increase in reliability that is obtained when more measurements are taken (and aggregated) from the same construct. Reliability is the ...
1
vote
0answers
22 views

How to use Cronbach's Alpha?

Is it appropriate to use Cronbach's Alpha for nominal variables? I have 20 nominal variables (two dimensions with 10 questions each) that ask about the characteristics of work performance. The ...
0
votes
0answers
46 views

Maximum likelihood estimate of θ

Here's the question im trying to solve. Correct me if im wrong please. S(360) means the deaths that occured in the time period of 0 to 360 hours. Does S7 means that there will be seven deaths during ...
0
votes
0answers
7 views

sample size calculation for intraclass correlation coefficient (ICC)

I want to design a study when I'll be comparing three measurements of a variable in imaging and then compute an ICC as a measure of agreement since I'll be obtain a continuous measurement. I read ...
0
votes
0answers
13 views

System Yield in Reliability

Let $Y(t)$ be the system yield function: \begin{equation} Y(t) = \int_0^\infty [v(t) - c(t)]dt, \end{equation} where $v(t)$ is the revenue at time $t$ and $c(t)$ is the cost at time $t$. The ...
0
votes
0answers
24 views

Is bootstrapping the correct way to prove reliability/unreliability of a statistic?

I have a sample of which I calculated an average, say m (the sample size was calculated using a confidence of 0.95 and error of 0.02; see RaoSoft calculator. So the population average should fall ...
0
votes
0answers
6 views

Can reliability and validity assessments be considered as required constituent of the “Measurement part” in Structural Equation Modeling?

I wonder if any type of reliability and / or validity analysis can be considered as an element of the Measurement part in SEM. For example, when computing Cronbach's alpha or the AVEs of an instrument,...
1
vote
0answers
29 views

Platt Scaling vs Isotonic Regression

I am learning classifier probability calibrations and have calibrated an eleastic net model using both Platt scaling and isotonic regression. As you can see in the attached image Platt scaling (on the ...
0
votes
0answers
9 views

Does the quoted line mean that the 6 cars would have gained more miles? If not, what does it mean?

I am reading Jayant Deshpande's "Life Time Data: Statistical Models and Methods" book and while reading about Right Random Censoring, I read about this example. Example of Random (right) Censoring ...
1
vote
0answers
9 views

Measuring Validity and reliability

Say you have an independent train ticket system where automatic refunds would be issued in case of a delayed arrival. To determine the arrival time of the train at a certain station you collect data ...
1
vote
1answer
35 views

Reliability in R

I'm having trouble understanding the difference between omega 1 and omega 3 in reliability function in the semTools package in <...
0
votes
0answers
8 views

Custom Zero-Fail Data Analysis Method

I'm trying to analyze products with zero-failure and get reliability values. I've looked into other methods here. Some of my products have very few observations and some have several hundred. I've ...
1
vote
0answers
26 views

Emergency Use Auth (EUA) for SARS-CoV-2 Tests

Problem: After the EUA for SARS-CoV-2 tests, we have a diagnostic test that passes both of the following criteria: All first five true-negative samples each produce a negative test result. All ...
1
vote
0answers
29 views

How to test the disturbance effect of a measuring instrument?

I am trying to think of an experimental design that allows me to test to what extent a measuring tool (automatic counting unit) alters (by restricting the entrance area) the measured quantity (number ...
0
votes
0answers
14 views

R: Reliability in higher order models

when modeling the big five using a higher-order model with factors on the second order and facets on the first, how can I calculate reliability with McDonald's ω in R? Is it even necessary? ...
0
votes
0answers
11 views

Difference between inter-itemcorrelation and Cronbach's alpha

What exactly is the difference between inter-itemcorrelation and Cronbach's alpha? Is the one better than the other when there is, for example, a lower amount of items (e.g., 4 items)? Thank you!
0
votes
0answers
16 views

Bland-Altman analysis for repeated measure across time with one method of measurement

Just like to seek some clarification. As far as I understand and have read "The Bland–Altman method calculates the mean difference between two methods of measurement (the ‘bias’), and 95% limits of ...
0
votes
0answers
28 views

Gamma to Beta: Successes and Failure Proportions?

I have a question after reading this article on the relation between Gamma and Beta distributions and this article on the intuitive understanding of the Beta Distribution. Question As the Gamma ...
2
votes
1answer
40 views

Inter rater reliability with multiple raters, multiple categorical variables with different levels

Apologies if the answer to this question exists elsewhere, but I haven't found anything quite matching what I'm looking for. I have a dataset wherein 3 raters independently assessed 10 subjects by ...
0
votes
0answers
30 views

AVE and CR in JASP?

I'm new to the software JASP and need some help. Is it possible to calculate average variance extracted (AVE) and composite reliability (CR) in JASP? I need them in order to test validity and ...
0
votes
0answers
17 views

Internal consistency: Alternative measures for unidimensional scales with a small number of items?

I use r and analyze a scale of 11 items, theoretically assuming one global factor and three group factors. Since a CFA revealed numerous correlated errors and due to the criticism of cronbachs alpha (...
2
votes
0answers
29 views

comparison between two measurement devices

I have two measurement devices accelerometer A and B. The measurement is the step time of a person walking in a corridor. A is the gold standard. I want to check the validity and reliability of B. how ...
1
vote
2answers
25 views

Reliability of a multistate item

I have a question and appreciate your time and input to give me some hints. If a system could stay in several states ranked based on the performance of the system such as (state 8 - perfect state, ...
0
votes
1answer
26 views

Inferential interpretation of Kappa reliability

I have obtained a Fliess' Kappa interrater reliability of $37.5\%$ from $90$ randomly selected raters rating $100$ randomly selected, binary items ($Yes/No$). Question: Is the following ...
1
vote
0answers
16 views

Can I use G theory for this case? How? Comparing Waldorf and traditional schools

I have my data in the format of: In my study I examined 20 waldorf student and 30 traditional student using concept map. Concept maps can be scored by different scoring methods. How can I determine ...
2
votes
1answer
35 views

Alternative to Cronbach's alpha if $n$ is too small for McDonald's omega?

I have a very high Cronbach's Alpha of $0.966$ (and composite reliability of $0.969$). My questionnaire has a 1-factor-structure, which makes it compatible with Cronbach's Alpha, but it has 22 items. ...
0
votes
1answer
12 views

How to calculate composite score with a 2-factor scale

Suppose I have a scale with four items meant to measure, say, happiness. I ran a confirmatory factor analysis and the model fits well with two factors and let's say that the two factors are 1) ...
0
votes
0answers
12 views

Error-bars on split half reliability

Say I administer a test to 30 students. I randomly divide the test questions into two parts and score each half for each student. I then calculate the correlation coefficient between scores on each ...
0
votes
0answers
9 views

Estimate the reliability in incomplete data

I have historical data (failure point) for a number of physical servers. These data only consist of the failure observed in 30 days. And, we know the lifespan of the physical server is around 3 to 5 ...
0
votes
0answers
14 views

Intrarater reliability for content analysis having multiple categories assigned to one case

I have a specific qualitative dataset of 130 reddit posts, from which I have 9 specific codes (categories) emerged. Now I would like to check my reliability by coding the same dataset with these codes ...
0
votes
0answers
11 views

What is a good measure of interrater reliability when 1 rater is predetermined and 1 rater is randomly assigned?

I have ordinal data with paired ratings: 1 a teacher self-score (actually a score of a student's work, but it is used to gauge learning growth) and 1 from a randomly assigned rater. The randomly ...
0
votes
1answer
71 views

Pre-test/Post-test Reliabilty

I calculated Cronbach's α to estimate reliability using both pre-test and post-test scores before and after a training. I fully expected the pre-test reliability estimates to be low, but what could ...
2
votes
1answer
77 views

What drives the test reliability of a composite based on three components?

Situation: A three part ability test with multiple-choice, essay, and oral components. I have scores for each of 110 test takers on each part and overall. The three parts are unequally weighted in ...
0
votes
1answer
54 views

Calculating the Expected life of network with hazard functions

I have been asked to solve some questions in the statistical program R. First, I want to find the expected life length (by numeric integration) of a network with 3 parallel components (T1,T2,T3) and ...
0
votes
0answers
27 views

Finding stability in a time series of success/failure data — Mixed Model Random Effects

I have a very large data set of success/failure data for hundreds of subjects. The chance of success is rather small but does/should stabilize over time. Subjects have varying levels of observations/...
0
votes
0answers
18 views

Validity and Reliablity Multiple Choice Question Questionnaire

I gave questionnaires to 7-8-year-old children to answer questions about oral health knowledge. I've already collected the data but I'm still confused with validity and the reliability in SPSS. I ...
2
votes
1answer
52 views

What transformation can I use to enable a relatively unbiased meta-analysis of $\alpha$ reliability?

I want to meta-analyse reliability estimates. For simplicity, assume the reliability estimate in question is Cronbach's $\alpha$ (aka Guttman's $\lambda$-3). Various packages exist to do meta-analyses ...
4
votes
1answer
341 views

Difference between empirical and marginal reliability of an IRT model

I am using the mirt library in R to fit an instrument (binary responses) comprising two dimensions. In the mirt documentation are mentioned two types of reliability. I would like to ask what is the ...
0
votes
0answers
23 views

Generalizability study of subscaled rating instrument

I have a school rating ($1$ thru $4$) instrument that consists of $9$ subscales (e.g., classroom instruction, school management etc.). Under each subscale, I have $6$ items on which rating occurs. I ...
0
votes
0answers
16 views

Cronbachs alpha interpretation - what does var.r and med.r mean?

i have used the psych::alpha to find the cronbachs alpha values, but i cannot find any interpretation for some parts of the output, namely: ...
0
votes
1answer
23 views

Is it always needed to do a validity test for variables?

Let's say I have one variable called "perceived image", which is measured with 6 items (questions) that I adapted from previous literature. If I have a Cronbach's alpha > 0.7 (for reliability), do I ...
0
votes
0answers
25 views

How do I combine the categories of a classification system to find the maximum reliability?

I described my study design in another question. Briefly, I recruited 16 unique raters to classify 35 items according to a seven-category classification system. I measured the interrater reliability ...
2
votes
0answers
54 views

Correcting for reliability scores for Hedges g

I've received a revise and resubmit on my meta-analysis investigating the effects of a positive psychology intervention on depression and anxiety symptoms. I used Hedges' g as the effect size within a ...
0
votes
0answers
17 views

TTF and Relibility data form

Assume the car engine has 4 failure in one month. Example: Failure at day 10, 15, 23, 29. Is TTF maintained after each failure as shown in the following table, or it is increasing? How to calculate ...
0
votes
1answer
21 views

Which distribution or process should be used for wearout reliability modeling?

When modeling the reliability of a system, it is usual to use exponential distribution to model errors that occur randomly throughout the system's useful lifetime (the middle part of the well-known ...
0
votes
0answers
11 views

inter-observer reliability

I want to test the inter observer reliability between myself and one other observer on three different groups of data. I am unsure hat the best statistic is to go with the data I have collected: 1) ...
0
votes
0answers
29 views

Reliability estimation for questionnaire - is Cronbach's a applicable?

I am fairly new to scale analysis and I need to estimate the reliability and validity of a 5 item scale (5 questions, 2 have 4 choices abcd and 3 have 2 choices ab). The scale is supposed to be ...

1
2 3 4 5
9