The reliability tag has no wiki summary.
1
vote
1answer
33 views
Do you always measure reliability through the Cronbach's alpha coefficient? Or is there another way? (multiple linear regression)
I started doing a quantitative research for my thesis without any previous SPSS or proper statistic experience, and the first thing my professor told us in the spss workshop is that you start by ...
0
votes
0answers
26 views
interrater reliability for multiple variables
i and another coder want to assess our agreement.
our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted ...
7
votes
2answers
445 views
What distribution does my data follow?
Let us say that I have 1000 components and I have been collecting data on how many times these log a failure and each time they logged a failure, I am also keeping track of how long it took my team to ...
0
votes
0answers
27 views
low reliability
In my data set, there are two scales; one scale has 33 items and other has 6, and my sample is 100. The scales' reliability low and negative. What should I do to improve reliability.
0
votes
0answers
37 views
How to assess Inter-rater reliability of frequency data with two raters using SPSS?
I am doing an observational study on the frequency (number of times) a particular person enters a shop. This study is being carried out by two different observers. I now want to do an inter-rater ...
1
vote
2answers
75 views
What factors do I have to consider when deciding if creating a composite score using four t-scores is defensible psychometrically?
Edit:
I've conducted an item analysis on four variables - two quizzes and two psychological attributes, namely emotional intelligence and happiness. Its a within-subject design with students as ...
3
votes
1answer
57 views
Validation of a scale for a different population (CFA)
I'm currently adapting a coping questionnaire which has been widely used in sports to be used with a different population (musicians), so I need to address its reliability and validity with this new ...
1
vote
0answers
211 views
Computing Cronbach's Alpha and Intraclass Correlation coefficient from an lmer model
I am trying to calculate the Intraclass Correlation for a rater study using R and the library lme4 and the function lmer. The data has the following design: The same 6 raters (at least 4) are rating ...
0
votes
1answer
79 views
Is optimizing α for partitions of a scale a sensible method for factor analysis? (was: relation between consistency and unidimensionality)
Cluelessness Disclaimer
I'm a statistics noob so if at all possible, please don't stone me. Also write slowly and in simple terms.
I'm wondering what the relation between the internal ...
0
votes
0answers
45 views
Reliability of a rating scale
My aim is to evaluate the reliability and validity of a new observation rating scale (likert-type, with points 1 to 5) of items, representing different behaviours. The sample does not meet parametric ...
1
vote
0answers
90 views
error variance calculations for reliability analysis of a composite metric in HLM
I am trying to determine how to obtain within-group variance for a composite measure based on a set of (weighted) proportions.
I have 50 groups being compared on 8 proportion measures, with 4 ...
1
vote
0answers
63 views
What is the proper test for inter-rater reliability (three raters, ordinal data)
I need help for choosing a proper test for inter-rater reliability. I have three raters (experts) who have analysed the quality of images (10 images from different location). The data is ordinal ...
7
votes
2answers
209 views
Handling poor inter-rater reliability while minimizing the loss of data
Edit: While I do appreciate Peter Flom's suggestions and the subsequent discussion (and upvoted his answer), I am opening a bounty to solicit an answer which offers a specific, formal statistical ...
2
votes
0answers
50 views
Assessing the ability of two quantitative methods measuring the same thing
Suppose I have two scientific approaches to measuring the same physical quantity taking values between $[0,1]$. One (say method A) is generally accepted to be more accurate under certain conditions, ...
1
vote
0answers
47 views
Reliability ratings in Bayes' theorem
In his 1967 paper “A Theorem for Prediction” (Studies in Intelligence 11(4)), Jack Zlotnick discusses how some CIA experiments lead to a “modification” of Bayes' theorem, to account for unreliable ...
1
vote
0answers
41 views
Omega H asymptotic
In the omega function in the psych package for R, we have several outputs including Hierarchical omega, Asymptotic omega and Total omega.
I was reading about the outputs, but I find nothing about ...
2
votes
0answers
58 views
How to assess stability of daily time series in sentiment analysis?
I developed a measure of "sentiment" and I have time based data and used the measure to derive a daily sentiment time series. I am looking for some way to establish reliability or maybe stability. For ...
3
votes
0answers
69 views
Modeling measurement error (reliability?) using repeated measurements
I have a measurement protocol that seeks to characterise the visual sensitivity of a subject. I would like to assess how reliable the measurements from this procedure are.
I have measured a small ...
3
votes
0answers
629 views
For intraclass correlation coefficients (ICC), interpret both “single” and “average” measured ICC values for reproducibility test?
Say there's a new machine for use in testing a biochemical marker's concentration in the blood, and we would like to know the reproducibility of this test.
We draw blood samples from 10 volunteers, ...
0
votes
2answers
1k views
What am I supposed to do with negative Cronbach alpha
I have gathered data from 130 questionnaires and I have 30 variables.
Cronbach alpha is badly negative and I have Checked every thing I knew. But I saw no change in result.
please help me
SPSS file ...
1
vote
0answers
255 views
How to validate a survey when using Likert scale?
I have developed a very simple survey to measure the attitudes of students toward science fairs. It makes use of the Likert Scale and Open Ended Questions. Do you have a recommendation for the type ...
2
votes
1answer
88 views
Basic Inter-rater reliability and type of data
I am looking to find out the type of data I have and the ideal test to use calculate inter-rater reliability.
Research question: Functional Movement Screening: How reliable are we in our ...
2
votes
0answers
109 views
How to calculate inter-rater reliability for just one sample?
I'm trying to compute a reliability rating for potentially malicious emails. Essentially a system where multiple people will view a suspect email, and assign it a "malice score". The idea is for ...
1
vote
0answers
54 views
Reliabilities or factorial analysis first?
I've computed a scale with several theoretically defined factors. Now I am doing a factorial extraction to see if EFA discovers the same dimensions as in theory. My question is:
Do I need to check ...
1
vote
1answer
638 views
Which statistical method to use in survey-based quantitative research?
I hope you can help me because at the moment I am confused and helpless. I am final year PhD student, and my supervisors asked me to find out statistical methods other than exploratory (percentage of ...
0
votes
0answers
47 views
How to assess mean estimate reliability?
Disclaimer: I am trying to fresh up my statistics from basic courses in the uni, so undeniably this is a very fundamental question, I hope you can keep that in mind in your answers.
Background: Given ...
4
votes
1answer
136 views
Internal consistency reliability in item response theory models
How can internal consistency reliability of a test and of individual test items be quantified in Item Response Theory models? I know I can resort to Classical Test Theory, Cronbach's alpha, and other ...
0
votes
2answers
278 views
Interpreting the results of my Cronbach alpha test
I have a survey which contains 13 variables in total. For each variable I asked 3 similar questions in order to test the internal consistency. Now I performed 13 Cronbach's alpha tests, one for each ...
0
votes
0answers
227 views
Is it necessary to calculate Cronbach’s alpha for each sub-test?
I have a data set that includes two sub-tests, each measured using a Likert scale.
I want to test the reliability.
Is it necessary to calculate Cronbach’s Alpha for each sub-test? Or do I have to ...
0
votes
0answers
130 views
Sample Size determination - cross over study with repeated measures
I am working on the sample size determination for cross-over study. The aim of this study is to measure reliability of the measurements of outcome variable (continuous). For example, There will be 50 ...
1
vote
0answers
81 views
Intrarater reliability
I would like to estimate the intrarater reliability in a MRI study. Different regions of interest were manually traced with a mouse-driven cursor in order to estimate their volumes. All data were ...
3
votes
2answers
175 views
Inconsistency of findings between Kappa and ICC for IRR study
I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects:
...
2
votes
1answer
258 views
Omega vs. alpha reliability
I wonder if someone can explain what is the main difference between omega and alpha reliabilities?
I understand an omega reliability is based on hierarchical factor model as shown in the following ...
1
vote
0answers
224 views
Assessing scale reliability: how to get the p-value for corrected item-total correlation?
I assessed the internal reliability of a self-created scale with eight items (N total = 150) by calculating Cronbach’s α. It appears that one item correlates low with the overall score of the scale ...
1
vote
1answer
691 views
Cronbach's alpha continuous and dichotomous data in SPSS
I have one question about using SPSS to calculate Cronbach's alpha of a measure that draws on two types of items: one yields ordinal data and the other dichotomous. The ordinal data are produced ...
1
vote
0answers
271 views
How to report Cronbach's alpha for referent-shift scale (before or after aggregation?)
I have a study variable that is measured at the individual level and then aggregated and to the group level. My hypothesis testing is conducted at the group level.
I used an 8-item referent-shift ...
3
votes
1answer
123 views
Rule of thumb reliability of scales for using manifest variables in path analysis?
I am using path analysis to test hypotheses in my study. I have been told by a few people in passing that I can use manifest variables rather than latent variables as long as my reliabilities are ...
2
votes
0answers
72 views
Testing graded response model thresholds for significance?
I have a data set in which two raters have each rated N samples using a 5-point ordinal rating scale. My primary interest is in whether these two raters make significantly different use of those ...
0
votes
0answers
72 views
How to show that method X is more reliable than method Y?
I compute a statistic s using two methods, X and Y. In order to measure the reliability of X and Y, I found split half reliability measures (alpha) using a bootstraping technique. So, for each of ...
2
votes
3answers
166 views
Methods for estimating the survival function
I want to estimate the survival function $S(t_0)=1-F(t_0)$, at a specific point $t_0$ where $F$ is a continuous cumulative distribution function, based on an uncensored sample $x_1,..,x_n$ and a ...
7
votes
1answer
210 views
What am I measuring when I apply a graded response model to the “Hunting of the Snark” dataset?
In another question, I asked about the statistical validity of StackExchange's "Hunting of the Snark" dataset, and whether or not we could draw any conclusions from its results. I measured a few ...
19
votes
2answers
480 views
Is Joel Spolsky's “Hunting of the Snark” post valid statistical content analysis?
If you've been reading the community bulletins lately, you've likely seen The Hunting of the Snark, a post on the official StackExchange blog by Joel Spolsky, the CEO of the StackExchange network. He ...
4
votes
1answer
845 views
How to reduce number of items using factor analysis, internal consistency, and item response theory in conjunction?
I am in the process of empirically developing a questionnaire and I will be using arbitrary numbers in this example to illustrate. For context, I am developing a psychological questionnaire aimed at ...
2
votes
0answers
407 views
Can I use a measure when it has low reliability?
I would like to use a 5-item subscale to measure the effects of a behavioural intervention program on the use of behavioural strategies. I obtain significant results when I analyse it in models ...
7
votes
1answer
641 views
How to perform inter-rater reliability with multiple raters, different raters per participant, and possible changes over time?
Participants were rated twice, with the 2 ratings separated by 3 years. For most participants the ratings were done by different raters, but for some (< 10%) the same rater performed both ratings. ...
1
vote
1answer
777 views
Is it plausible to get a Cronbach's alpha of .85 with only two Likert-type items?
I was reading an article that reported a Cronbach's alpha of .85 for a two-item scale. The scale was measuring self-perceived competence. Each item was on a 7-point Likert-type scale.
Intuitively, ...
4
votes
2answers
2k views
Is it acceptable to use Cronbach's alpha to assess reliability of questionnaire composed of categorical and conditional items?
Background: I have data on a questionnaire made up of categorical factual items. Some of them are binary and others have more than two categories.
For example:
...
27
votes
1answer
1k views
Computing Cohen's Kappa variance (and standard errors)
The Kappa ($\kappa$) statistic was introduced in 1960 by Cohen [1] to measure agreement between two raters. Its variance, however, had been a source of contradictions for quite a some time.
My ...
1
vote
2answers
218 views
Split-half reliability on binary scores
I'm very new at statistics, so excuse me if my questions are to simple or obvious.
I have a dataset with response times for two conditions. These response times were converted into a binary score of ...
4
votes
1answer
438 views
Interrater reliability in SPSS
I am trying to calculate interrater reliability in SPSS for both pre and post test of the same measure that is administered as part of a prison intake program. The measure has 20 items and then a ...
