A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki).

learn more… | top users | synonyms

1
vote
1answer
17 views

wondering about Negative alpha cronbach

I used a questionnaire that usually has a high Cronbach's alpha and that has a scale of 7 answers. I did a before and after (an intervention) evaluation and got a very very low alpha for the before ...
3
votes
4answers
80 views

Sample selection algorithms to ensure that training & validation sets are representative

Currently, I am encountering a question, which is how to selection representative samples (training set and test set, even validation set) from the whole data set? I would like build a classification ...
1
vote
0answers
3 views

How remove variables to a certain number from Cronbach's Alpha?

I'm looking to use Cronbach's Alpha for one of my independent variable from my questionnaire (it consist of 8 variables, meaning that 8 questions are testing for this one particular independent ...
1
vote
0answers
25 views

Is it ok to delete items when Cronbach's alpha is low?

I have used a questionnaire in my research that is showing a Cronbach's alpha of .52, even though it has shown good internal consistency in many previous studies. I can see that if I delete 2 of its 9 ...
1
vote
0answers
12 views

Choosing between two scales

I am investigating a scale of consumer ethnocentrism - CETSCALE. I do not have much knowledge in econometrics, scales, etc., however: a) research papers report that the standard 17-item scale is ...
1
vote
0answers
10 views

How to combine data from different papers to test a standard scale?

I am very new to statistics and econometrics, however, I have to investigate one scale that was used over 30 years. It measures customer ethnocentrism. I want to combine data from different studies ...
0
votes
1answer
16 views

Reliability test, Cronbach's alpha or else

I have a set of questions for my research on Importance Menu Design towards Customer Purchasing Decision. After running Cronbach's Alpha for reliability test, it turned out less than 0.5 These are ...
0
votes
0answers
14 views

Individual item reliability in factor analysis

Im currently testing the reliability and validity of the SERVQUAL model (measuring just the items for each dimension. Hence these items should load on just factor). My supervisor wants me to include ...
1
vote
0answers
14 views

Kappa (interrater agreement) alternative for count-based data which results in inflated agreement at lower counts?

I have a data set where four coders are rating 800 items on various attributes. This is achieved by reporting a count of the prevalence of each attribute, e.g. rater 1 thinks attribute A appears in ...
2
votes
0answers
19 views

Estimating and using reliability metric with the same data

I'd like to know if it is reasonable to use the same set of data to estimate and use a reliability metric from repeated measures? I am mostly interesting by the minimal detectable change (MDC). But I ...
0
votes
1answer
18 views

Given MTTF and Number of Items, how to calculate failing parts with Time?

I am wondering how to make use of MTTF Here is the situation, I am given an MTTF for an item type x and a certain demand for that item in the next 25 years, say 100 parts that will be in operation ...
0
votes
0answers
51 views

Should I use both paired t-test and intraclass correlation coefficient to examine the interrater reliability of an instrument?

As title, I am assessing the interrater reliability of a questionnaire (actually a Barthel Index (BI)) performed by four assessors (A,B,C,D) on about 25 test subjects. I plan to examine the IRR from ...
0
votes
0answers
43 views

Using Cronbach's alpha to test reliability of a questionnaire

I administer a questionnaire to managers asking them to what extent they agree / disagree that they perform certain functions (listed in the questionnaire) on a scale of 1-5. The sample functions are ...
0
votes
0answers
19 views

Non-normal data (ceiling effect) and intraclass correlation

I am wondering if there is any way of doing an intraclass correlation, two-way, mixed, with non-normal data (in this case a ceiling effect)? Alternatively, what the interpretation (if any) would be? ...
0
votes
0answers
23 views

Any reliability conclusion from many duplicate data points?

I'm trying to draw reliability conclusion about a material under a tensile test. All of my data max out the sensor. The spec is 8 lbf and I'm maxing out a 50 lbf load cell. "Get a bigger sensor" ...
0
votes
1answer
20 views

When is a poor reliability index acceptable?

I have tested this particular scale and it turns out that it has low reliability index (a=4.5). I have already deleted two items but it only increased a bit by a=.67. My sample size is 25 as part of ...
2
votes
0answers
34 views

which statistic should I use for inter-rater agreement on nominal data, multiple responses

I have over 400 open-ended responses. The total has been coded by one rater, and a 10% identical sample of the total has been coded by 2 raters. There is a choice of 14 nominal codes, of which the ...
1
vote
1answer
37 views

Cronbach's Alpha low- significant results

One of my variables has a low Cronbach's Alpha. I continued to run an planned contrast to test my hypothesis and the results were significant. I understand that the results should be interpreted ...
0
votes
0answers
12 views

Effect of Violating Spearman_Brown Prophecy Formula Assumptions

The Spearman-Brown Prophecy formula is sometimes used to predict the reliability of the average assessment based on N raters using the interrater reliability calculated from the N sets of individual ...
0
votes
0answers
16 views

Reliability Measure for Unequal Subjects between Raters?

I am comparing two medical imaging analysis techniques, and I would like to assess the inter- and intra-rater reliability for using each method. I am planning the study design, and would like help ...
2
votes
0answers
96 views

Variance component analysis with lme4

I am beginner with lme4 and I am seeking some advice on how to carry a variance component analysis. The data come from a real-world scenario, not from a designed experiment, resulting in a complex ...
3
votes
0answers
91 views

What is the relationship between scale reliability measures (Cronbach's alpha etc.) and component/factor loadings?

Let's say I have a dataset with scores on a bunch of questionnaire items, which are theoretically comprised of a smaller number of scales, like in psychology research. I know a common approach here ...
0
votes
1answer
294 views

Can ICC values be negative?

I am calculating the ICC as per the method of Shrout-Fleiss (1979), using the (3, 1) model in which the judges are fixed as are the targets. Some of my ICC values are negative. Is this to be ...
0
votes
1answer
44 views

Measuring Cronbach Alpha for very large number of items

How can I calculate Cronbach Alpha for two tests that I have administered to 165 people? One test contains 110 items and the other contains 67 items. The scoring is done on the hard copies and the ...
3
votes
1answer
71 views

Why change items which don't change Cronbach Alpha?

I am doing peer review on a paper submission for a journal, and I am somewhat out of my depth. The authors are developing a psychometric scale. They did a pilot study with 45 items. They measured ...
7
votes
2answers
921 views

How is the Spearman-Brown prophecy formula affected by questions of differing difficulties?

How are the results of the Spearman-Brown prophecy formula affected by having test questions of differing difficulties or raters who are easy or hard graders. One respected text says the S-B is ...
0
votes
0answers
5 views

Reliability estimate for each item (comparing raters to each other)

I am working with a 15 item scale of pre-school teaching. Five observers observed 58 teachers. Its a complete crossed matrix i.e, all raters observed all teachers. In an attempt to weed out the weak ...
0
votes
0answers
6 views

Assessing test-retest reliability of a questionnaire [duplicate]

I am testing the test-reliability of a newly developed questionnaire over two time points. The study involves parents answering about their child over the two occasions. Would intraclass correlation ...
0
votes
0answers
19 views

Cronbach's Alpha for open questions

In the questionnaire I asked respondents from two countries how many job offers they received from 5 sources in the last 6 months. There are 5 questions - one for each source. It is an open question, ...
6
votes
1answer
107 views

Item reliability or fit in IRT-based adaptive tests?

My background is in machine learning and statistics, but I am relatively new to psychometrics and testing. Nearly all of the literature I've found on item reliability refers to Chronbach's alpha or ...
1
vote
0answers
19 views

can the same item be used in multiple constructs?

I am constructing a survey with around 10 multi-item scales. My piloting indicates that one of the most important multi-item scales (COP) has very poor reliability (.49), but there is an item in ...
4
votes
1answer
65 views

How to compare Cronbach's alpha when the grouping variable is continuous?

I want to test if the $\alpha$ for a scale is dependent upon a personality test, which is a continuous variable. I understand that there are many methods to test if two $\alpha$'s are significantly ...
0
votes
0answers
38 views

Replication reliability analysis

does anyone know about replication reliability analysis? I'm conducting a research which is similar to the study below (measure inter- and intra-subject reliability): ...
0
votes
1answer
238 views

Shall I remove factors because of low Cronbach alpha level?

I've got 2 questions actually. I am adopting a scale from USA to Turkey; the original scale has six factors, but based on the results of factor analysis, I found that adopted one consists of 9 ...
0
votes
0answers
46 views

Statistical formula for intra-rater reliability

I designed an assessment rubric for a class assignment and wish to calculate my own intra-rater reliability. My goal is to use the rubric to evaluate the same ordinal data at two different points in ...
3
votes
0answers
58 views

How to measure the reliability of a consensus ranking (problem from Kemeny-Snell book)

Suppose that $k$ experts are each asked to rank a set of $n$ objects in order or preference. Let allow ties in the rankings. John Kemeny and Laurie Snell in their 1962 year book "Mathematical models ...
0
votes
0answers
18 views

Starting from available data, how do I test the reliability of a technique that generates a score?

I have hundreds of samples of papilloma which are caused by human papilloma virus 6 or 11. There are many ways to determine which HPV is in a specimen. Is it 6 or 11 or a combination of both? We chose ...
0
votes
0answers
138 views

Sample size calculation to demonstrate reliability of physical test method

I'm preparing a procedure for validating attribute physical test methods (e.g. vision system) by demonstrating the reliability of the test method to detect known good/bad samples.I have sourced the ...
10
votes
2answers
246 views

Identification of useless questions from a questionnaire

I'm developing a questionnaire. To improve its reliability and validity I want to use statistical methods. I want to eliminate questions whose answers are always the same. This means that nearly all ...
0
votes
2answers
441 views

Reliability of a network

Consider the communication network shown in the figure below and suppose that each link can fail with probability p. Assume that failures of different links are independent: All links ...
2
votes
0answers
42 views

Repairable system and the sum of GEV random variables

We know that $X\sim {\mathrm {GEV}}(\alpha ,\beta ,0)$ and $Y\sim {\mathrm {GEV}}(\alpha ,\beta ,0)$ then $X+Y\sim {\mathrm {Logistic}}(2\alpha ,\beta )$. I am wondering, what will be $X+Y+Z$ ...
4
votes
4answers
433 views

If a measure is valid (but not necesarily reliable), can it be consistently replicated?

I had this question on an exam, and I was positive that the answer was A. The Beck Depression Inventory is a scale intended to measure depression levels, with higher scores indicative of higher ...
0
votes
1answer
102 views

Weibull analysis [duplicate]

I need to get the maximum likelihood estimate of a two-parameter Weibull distribution. I have tried it on my own but everything tends towards using a Newton-Raphson iterative method which I am not ...
0
votes
0answers
24 views

How can we measure between-experimenter reliability for experimental data?

Description of my situation: I will run an experiment in a few weeks with a total sample size of N = 70. Ideally I would prefer to have only one experimenter to collect all the data but I have to do ...
0
votes
0answers
30 views

Drawing an ROC curve from one point

First, thanks for the wonderful site. Second, I am trying to use an ROC curves to check the reliability of some nondestructive testing of concrete slabs. As you may know, the Non-destructive ...
1
vote
0answers
42 views

Optimising test re-test reliability through randomly generating hypothetical repeated trials

I have had a look around the forum, and despite some similar questions on measurement error, no one appears to have asked this question specifically. I have developed a test where a small group of ...
0
votes
0answers
117 views

Assessing questionnaire reliability using test-retest measures

I am an undergraduate student with not much knowledge regarding statistics. I was hoping you would be able to help me by answering a question. I am currently checking the reliability and validity of ...
2
votes
1answer
32 views

How to create an “alignment” variable from mutual rating variables

I want to create an "alignment" variable out of two other variables. Meaning, that if two raters (say manager and employee) rate each other similarly, alignment is high, but if they rate each other ...
0
votes
2answers
94 views

Do reliability and construct validity have same technical(statistical) assumptions?

It appears that reliability and construct validity have a similar backing in terms of statistics. Is it true ?
1
vote
1answer
30 views

Reliability measure with no single item scores

Let's say, I have an intelligence test made of 60 questions (items) which produces results for two intelligence constructs on a 0-100 scale (for instance crystallised and fluid intelligence). I only ...