A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki).

learn more… | top users | synonyms

1
vote
1answer
24 views

An invalid questionnare with valid resultant scores?

In what situation, a questionnaire could be invalid but its resultant scores could be valid? I received a comment from a journal article reviewer that says: In keeping with the preferred ...
2
votes
2answers
28 views

Should I use paired t-tests or ANOVA when comparing multiple variables from 2 tests?

I had 50 subjects perform two 30-minute cycling time trials on separate days. I would like to compare the means of several variables from each of the two tests: Cadence, Heart Rate, Power Output, and ...
0
votes
0answers
8 views

Negative Reliability for Pre Assessment BUT Okay Reliability for Post Assessment

I have gathered data from only 6 participants participating in a program. Before running my analysis, I wanted to check my reliabilities. The 2 item PRE assessment has a negative alpha. However, when ...
0
votes
0answers
26 views

sample size for test-retest reliability

I am interested in estimating the following: test-retest reliability for ordinal items; test-retest reliability for interval-level scale scores consisting of the items; and the optimal sample size ...
0
votes
0answers
10 views

How many number of items should i include in my test?

I have designed two post test ,a recognition test and a production test.I have inserted 18 items in each test. As I sifted through articles , I 've read that we should include at least 30 items to get ...
0
votes
0answers
25 views

Examining Validity and Reliability of a new questionnaire (Likert scale)

I need to examine the psychometric properties of a newly developed questionnaire. There are 22 items with a 5-point Likert Scale. The designed questionnaire includes two main scales. How do I ...
0
votes
0answers
13 views

The Effect of Unreliable Observed Variables in Factor Analysis (FA)

My question relates to the impact of unreliable observed variables in common factor analysis. Does common factor analysis treat the unreliable portion of a variable's variance as a unique variance? ...
0
votes
1answer
43 views

What is the statistic to calculate Cronbach's alpha for Polytomous items in R

Does anyone know, what is the function which can calculate Cronbach's alpha for polytomous items with different categories in R? Thank you
0
votes
0answers
20 views

PCA reduction and low-reliability components

I'm working on a survey with 288 observation in total (108 complete answers used) and around 200 variables. I'm working on reducing those number using Principal Components Analysis, using R. Suppose ...
0
votes
1answer
37 views

How to know if my data is balanced or imbalanced for an ROC curve analysis?

I am doing a research on the reliability of different models in detecting hidden defects in a test specimen. I have made a test specimen with defect prevalence about 25% (12 positive out of 49 total ...
1
vote
1answer
65 views

Calculating Cronbach's alpha: Same scale, multiple instances

I'm working on data analysis for my dissertation, and I'm banging my head against the wall when it comes to Cronbach's alpha. I just completed a pretest; after data cleaning, I have 119 valid and ...
0
votes
0answers
20 views

Is it possible to incorporate on/off cycles into a product's failure rate prediction along with its field failure rate and test data?

If I have a electronic module whose failure rate I am calculating using the chiinv() function in Excel, and the following data available to me: -test failures and test hours for the module and the ...
2
votes
2answers
129 views

Reliability in IRT Style

Reporting 'reliability' in IRT is more to 'informations' obtained and a bit different from a classical way using such as Cronbach alpha and KR20. Therefore, how to report the test reliability in ...
0
votes
0answers
13 views

f-test in context in scale reliability

In SPSS I have Scale->Reliability test. I used it for finding Cronbach alpha for my scale (11 items overall). However, in the same window in SPSS there is a checkbox "ANOVA TABLE -> F-test". I do not ...
2
votes
2answers
44 views

Cluster analysis as a preliminary analysis

I want to produce four groups (high/high, high/low, low/high and low/low) using two continues variables and compare these groups in terms of a few dependent variables. I know that cluster analysis ...
0
votes
0answers
13 views

Converting split-half correlations into coefficient of determination

It's my understanding that after finding the correlation (r) for a split-half correlation, you should convert that into a more accurate number via the Spearman-Brown formula, 2r/(1+r). Now I want to ...
2
votes
0answers
9 views

Split thirds correlation

You hear about split-half correlations often as a measure of reliability. But is there a reason that split-third correlations (or split-fourth, split-fifth, etc.) are never used? Shouldn't they ...
0
votes
0answers
31 views

Fleiss' Kappa alternative for inter-rater reliability with multiple answers

I have a dataset where researchers assign a binary yes/no answers to cells in a matrix. For each row more than one, but at least one, 'x' is allowed. For example, researchers 1 may code: ...
0
votes
1answer
298 views

Cronbach's alpha in R

I am pretty new to Cronbach's Alpha, so I have a question about how to find it using R. As far as I am aware, Cronbach's alpha is a measure of internal consistency, but to use the alpha() function in ...
0
votes
1answer
28 views

Minimum number of tested patients to have a reasonable ROC curve [closed]

What are the minimum number of tested patients and the acceptable prevalence percent required to have a reasonable ROC curve? for example, can I test a total of 16 patients, 5 are diseased and 11 are ...
0
votes
1answer
56 views

Is 0.40 Alpha score (KR-20) acceptable for a true-false test?

I have 10 true-false-Idont know test items. I codded correct responses as 1, incorrect ones and dont knows as 0. I got 0.40 alpha score (or KR-20). Do you think that this score is acceptable for such ...
1
vote
1answer
33 views

High reliability imply difference?

Suppose a test has scores between $0$ and $50$. Alice scores a $34$ and Bob scores a $42$. Does a high Cronbach's $\alpha$ imply that the Alice and Bob's scores are statistically significantly ...
0
votes
0answers
12 views

Intraclass correlation with a sparse rater-subject matrix

I'm working on a problem pertaining to the reliability of faculty evaluations of students. My dataset contains ratings of ~160 students given by ~50 faculty members. Most students are rated by 2 ...
3
votes
3answers
124 views

Probability that node A is connected to another node

Example: I have nodes A, B and C. A is connected to B and C. B is also connected to C. The link between two nodes have a probability to fail. For the link between A and B, the probability is pAB ...
1
vote
1answer
66 views

Expected lifetime of a device with two parts each having spares?

Consider a device with two parts : (1) and (2). Part (1) has 2 spares and part (2) has one spare. Lifetime of part (1) and its spares have iid exponential distribution with rate lambda. Lifetime of ...
1
vote
1answer
35 views

wondering about Negative alpha cronbach

I used a questionnaire that usually has a high Cronbach's alpha and that has a scale of 7 answers. I did a before and after (an intervention) evaluation and got a very very low alpha for the before ...
3
votes
4answers
184 views

Sample selection algorithms to ensure that training & validation sets are representative

Currently, I am encountering a question, which is how to selection representative samples (training set and test set, even validation set) from the whole data set? I would like build a classification ...
1
vote
0answers
20 views

How remove variables to a certain number from Cronbach's Alpha?

I'm looking to use Cronbach's Alpha for one of my independent variable from my questionnaire (it consist of 8 variables, meaning that 8 questions are testing for this one particular independent ...
1
vote
0answers
17 views

Choosing between two scales

I am investigating a scale of consumer ethnocentrism - CETSCALE. I do not have much knowledge in econometrics, scales, etc., however: a) research papers report that the standard 17-item scale is ...
0
votes
2answers
75 views

Reliability test, Cronbach's alpha or else

I have a set of questions for my research on Importance Menu Design towards Customer Purchasing Decision. After running Cronbach's Alpha for reliability test, it turned out less than 0.5 These are ...
1
vote
0answers
29 views

Kappa (interrater agreement) alternative for count-based data which results in inflated agreement at lower counts?

I have a data set where four coders are rating 800 items on various attributes. This is achieved by reporting a count of the prevalence of each attribute, e.g. rater 1 thinks attribute A appears in ...
2
votes
0answers
19 views

Estimating and using reliability metric with the same data

I'd like to know if it is reasonable to use the same set of data to estimate and use a reliability metric from repeated measures? I am mostly interesting by the minimal detectable change (MDC). But I ...
0
votes
1answer
48 views

Given MTTF and Number of Items, how to calculate failing parts with Time?

I am wondering how to make use of MTTF Here is the situation, I am given an MTTF for an item type x and a certain demand for that item in the next 25 years, say 100 parts that will be in operation ...
0
votes
0answers
132 views

Should I use both paired t-test and intraclass correlation coefficient to examine the interrater reliability of an instrument?

As title, I am assessing the interrater reliability of a questionnaire (actually a Barthel Index (BI)) performed by four assessors (A,B,C,D) on about 25 test subjects. I plan to examine the IRR from ...
0
votes
0answers
91 views

Using Cronbach's alpha to test reliability of a questionnaire

I administer a questionnaire to managers asking them to what extent they agree / disagree that they perform certain functions (listed in the questionnaire) on a scale of 1-5. The sample functions are ...
0
votes
0answers
47 views

Non-normal data (ceiling effect) and intraclass correlation

I am wondering if there is any way of doing an intraclass correlation, two-way, mixed, with non-normal data (in this case a ceiling effect)? Alternatively, what the interpretation (if any) would be? ...
0
votes
0answers
38 views

Any reliability conclusion from many duplicate data points?

I'm trying to draw reliability conclusion about a material under a tensile test. All of my data max out the sensor. The spec is 8 lbf and I'm maxing out a 50 lbf load cell. "Get a bigger sensor" ...
0
votes
2answers
31 views

When is a poor reliability index acceptable?

I have tested this particular scale and it turns out that it has low reliability index (a=4.5). I have already deleted two items but it only increased a bit by a=.67. My sample size is 25 as part of ...
2
votes
0answers
50 views

which statistic should I use for inter-rater agreement on nominal data, multiple responses

I have over 400 open-ended responses. The total has been coded by one rater, and a 10% identical sample of the total has been coded by 2 raters. There is a choice of 14 nominal codes, of which the ...
2
votes
1answer
68 views

Cronbach's Alpha low- significant results

One of my variables has a low Cronbach's Alpha. I continued to run an planned contrast to test my hypothesis and the results were significant. I understand that the results should be interpreted ...
0
votes
0answers
17 views

Effect of Violating Spearman_Brown Prophecy Formula Assumptions

The Spearman-Brown Prophecy formula is sometimes used to predict the reliability of the average assessment based on N raters using the interrater reliability calculated from the N sets of individual ...
0
votes
0answers
21 views

Reliability Measure for Unequal Subjects between Raters?

I am comparing two medical imaging analysis techniques, and I would like to assess the inter- and intra-rater reliability for using each method. I am planning the study design, and would like help ...
2
votes
0answers
125 views

Variance component analysis with lme4

I am beginner with lme4 and I am seeking some advice on how to carry a variance component analysis. The data come from a real-world scenario, not from a designed experiment, resulting in a complex ...
3
votes
0answers
156 views

What is the relationship between scale reliability measures (Cronbach's alpha etc.) and component/factor loadings?

Let's say I have a dataset with scores on a bunch of questionnaire items, which are theoretically comprised of a smaller number of scales, like in psychology research. I know a common approach here ...
1
vote
1answer
859 views

Can ICC values be negative?

I am calculating the ICC as per the method of Shrout-Fleiss (1979), using the (3, 1) model in which the judges are fixed as are the targets. Some of my ICC values are negative. Is this to be ...
0
votes
1answer
60 views

Measuring Cronbach Alpha for very large number of items

How can I calculate Cronbach Alpha for two tests that I have administered to 165 people? One test contains 110 items and the other contains 67 items. The scoring is done on the hard copies and the ...
3
votes
1answer
119 views

Why change items which don't change Cronbach Alpha?

I am doing peer review on a paper submission for a journal, and I am somewhat out of my depth. The authors are developing a psychometric scale. They did a pilot study with 45 items. They measured ...
7
votes
2answers
2k views

How is the Spearman-Brown prophecy formula affected by questions of differing difficulties?

How are the results of the Spearman-Brown prophecy formula affected by having test questions of differing difficulties or raters who are easy or hard graders. One respected text says the S-B is ...
0
votes
0answers
5 views

Reliability estimate for each item (comparing raters to each other)

I am working with a 15 item scale of pre-school teaching. Five observers observed 58 teachers. Its a complete crossed matrix i.e, all raters observed all teachers. In an attempt to weed out the weak ...
0
votes
0answers
8 views

Assessing test-retest reliability of a questionnaire [duplicate]

I am testing the test-reliability of a newly developed questionnaire over two time points. The study involves parents answering about their child over the two occasions. Would intraclass correlation ...