A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki).

learn more… | top users | synonyms

0
votes
0answers
5 views

Cronbach's Alpha - reliability

In psychometric tests, we sometimes are instructed to end the test after the child has failed a determined number of questions. Thus we end up with children who answered 30 questions and some who ...
0
votes
1answer
4 views

internal reliability scores below .70

When one is assessing the reliability of one instrument and the value is under .70 and you then eliminate the item/items that are mentioned that if eliminated would increase the reliability score, ...
0
votes
2answers
22 views

internal reliability

I have been searching for information on how to conduct internal reliability assessments but have been unsuccessful. Is there a book or website that can inform me on how to conduct these analysis? I ...
1
vote
0answers
25 views

Dealing with survival analysis data - reliability

I am totally new on Survival Analysis and my data is tricking me. I have, in excel, a list of failure's records informing date and hour of ocurrence, what failed/broke on that equipment and the cause ...
0
votes
0answers
23 views

Is my survey invalid now?

I'm running an online survey for my thesis. In order for the survey to be valid, I needed a minimum of 375 participants. And I have achieved this number. However, upon closer inspection I have ...
0
votes
1answer
43 views

Given a 1-Sided Lower Confidence Bound value, How to determine target failure rate

$R(t)$ = reliability = 1 - unreliability (probability that item is still operational for a given time t) unreliability = Cumulative Distribution Function (CDF) = (...
0
votes
1answer
37 views

What type of study?

I calculated inter-rater reliability for a medical test. We applied the test on both healthy people and people with a specific disease and then two raters interpreted the test. Can I call this study a ...
0
votes
0answers
14 views

Ordinal Alpha Reliability: Is Re-Ranking Appropriate?

I'm a researcher currently based in New Delhi where I'm helping adapt and validate a psychometric questionnaire. For this I need help with fixing the error in calculating ordinal alpha in R using the ...
2
votes
1answer
21 views

General validity/reliability of questionnaires

I have a general question concerning the reliability/validity of questionnaires. I'm working in Human Resources and we plan to conduct a workplace engagement survey. I attended a conference and a ...
0
votes
1answer
12 views

Inter-rater reliability simultaneously on multiple binary data

I have drug prescription records on 5 different drugs (A,B,C,D,E) on 2 different time points over a bunch of patients. 1/0 stands for use or not-use the specific drug, respectively. My questions are,...
1
vote
0answers
20 views

NVivo IRR Test of Content Analysis Data: How do I interpret each row of Kappas?

I am currently working on a research project that involves content analysis of secondary published data (corporate sustainability reports) by U.S. companies. In order to capture the data, I and two ...
1
vote
1answer
31 views

what am i supposed to do if realiability test not up to 0.7?

hye. i have gathered 147 questionnaires with 39 variables the requirement for my research department need me to make the reliability test up to 0.7 or else it consider not reliable. i have tried ...
1
vote
1answer
22 views

Intuition about f1 score

Let's say I have a data set where half of the data points are labelled as positive and half are labelled as negative. My task is to create a classifier which recognises when a sample from the dataset ...
0
votes
0answers
50 views

Estimation of the probability of the winner's curse

The increase in false-positive results arises primarily from an overestimation of effect sizes, leading to upward-biased effect sizes that often cannot be reproduced in follow-up studies (‘the winner'...
0
votes
0answers
12 views

Reliability of measurement scale of non-normal variables

I have 18 variables (measured in a Likert-scale) which should according to previous studies measure the same thing, let's say intelligence. I want to calculate the reliability of the sum variable of ...
0
votes
0answers
7 views

Formula to convert test-retest reliability into SD of difference

Wanting to work out the statistical power of a repeated-measures t.test. This requires an estimate of the SD of differences within pairs (the correlation between measures for a given person is what ...
0
votes
0answers
30 views

Principal Component Analysis: Scale reliability and validity

I'm using a popular psychometric scale, the Domain Specific Risk Taking (DoSpeRT) scale, that contains 30 items that measure an individual's risk taking capacity across five different dimensions: ...
0
votes
1answer
43 views

What to do with negative ICC values? Adjust the test or interpret it differently?

For my graduation thesis I am doing a study for the test-retest reliability of the tendon thickness of a particular muscle. So this study contains one rater and 70 subjects who have been tested at two ...
1
vote
1answer
17 views

Test Retest of highly skewed dichotomous variables

Does anyone have recommendations on how to evaluate test retest reliability with 95% CI (individuals took the same survey 1 week apart) on data that is dichotomous (yes = 1; no = 0) where the data is ...
0
votes
0answers
11 views

Access validity and reliability for hypotheses testing in prediction model

In my bachelor thesis I'm investigating a transaction data set of a retailer. I'm trying to find out indicators that predict a certain outcome, which is why I proposed several hypotheses for the ...
0
votes
0answers
17 views

1-Sided Lower Confidence Level Concept for Reliability (1-F(T))

For reliability (R) engineering discipline, reliability of an item is the conditional probability that the item will be functional within the mission duration at a given confidence level. so ...
0
votes
0answers
6 views

Is there a common recommendation on product lifetime declarations in terms of Reliability function?

I mean: if I read on the label that a specific product (car, household appliance, whatever...) is declared to have a lifetime of N years, does it actually mean that the Reliability function R(t), ...
0
votes
0answers
68 views

Cronbach Alpha Assumptions

I'm currently learning ho to use Cronbach Alpha in R. I have a scale with 7 items and 63 respondents. The aim is just to get a practical understanding of what Cronbach Alpha is. There are some nulls ...
0
votes
0answers
35 views

Calculate Cronbach's alpha for three observers

I have a dataset which contains 100 observations, and I asked three expert to describe each observation by one of the letters {A,B,C}. The following three tables shows a sample of the dataset with the ...
1
vote
1answer
42 views

Clear explanation of Cronbach's Alpha

I have difficulty in understanding Cronbach's Alpha's formula. I searched for google and StackExchange able to understand it. Even there are some topics trying to explain it intuitively, I could not ...
0
votes
0answers
10 views

Compute interrater reliability having different subsets of students rated by different subsets of raters

I would like to compute inter-rater reliability for a set of continuous items, having different subsets students assessed by different subsets of raters (please, see the picture). I would like to ...
2
votes
1answer
33 views

Cronbach's alpha vs. Average Item-Sum correlation?

I have heard that Cronbach's alpha is used to measure reliability of a test (to check if a set of items/question are measuring the same construct). How does the Cronbach's alpha compare with the ...
1
vote
1answer
12 views

Randomization scheme for rating images and assessing inter-rater reliability

Apologies for the basic question, but I'm new to the area and I'm working on a research project on rating images, and assessing the reliability of these ratings, with the following structure: There ...
0
votes
0answers
12 views

How to estimate the reliability of an exhaustive list of data?

I have a purported exhaustive list of beneficiaries under a social security scheme. My question is how large must be the sample of this population must be tested to check for fraud etc to derive an ...
0
votes
1answer
22 views

Intrarater reliability test for a single subject?

I have repeatedly measured the volume of the brain of 1 subject in an MRI study. Measurements were acquired by the same rater (myself) on three separate occasions resulting in three measurements of ...
1
vote
0answers
26 views

Effect of reliability on t-test

I have some problem with investigating the effect of reliability on the t-test. Basically it is assumed that lower reliability leads to a broader confidence interval or higher SD, which has a ...
0
votes
1answer
51 views

How can I calculate percent agreement and kappa for each category?

My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges ...
1
vote
0answers
37 views

Goodness of Fit and reliability of coefficients in Logistic Regression

I have logistic regression model with 9 independent variables. Two of them are numerical, the others are binary. Three of my coefficients are significant at 0.001, two of them are significant at 0.05. ...
1
vote
1answer
19 views

What to report for reliability analysis?

After completing data collection on a reliability analysis where multiple examiners sampled each subject three times, which value should I report for descriptive statistics? If intra-rater reliability ...
0
votes
1answer
31 views

Why is my Cronbach's $\alpha$ negative [duplicate]

I'm just learning about Cronbach's $\alpha$. I have recently tried to carry out this test. Having 80 completed questionnaires with 15 questions (variables), the questionnaire used the Likert scale 1-5 ...
0
votes
1answer
111 views

Inter-rater reliability measure with multiple categories per item [duplicate]

I have multiple raters providing categorical ratings. What makes this slightly different from other posted questions is that each rater can provide three, unique categorical ratings. Each rater had ...
0
votes
0answers
19 views

Effect of reliability on sample size

I asked a similar question here: Meta-analysis using experimental reliability data However as I did not get any feedback (probably for a good reason, which I would like to understand) I ask the ...
1
vote
0answers
19 views

Combining the standard errors of Cohen's Kappa computed for multiple rater pairs

In a study on intra-rater agreement with 8 raters, I have computed Cohen's Kappa for each of the raters and I estimated the standard error of each of the kappa values using the Jackknife method. To ...
0
votes
1answer
54 views

Normalization of values

How do I normalize a set of data? Let's say we are looking at measurements of diameter of the heart in women and men. Men will have bigger hearts, so error in measuring by the observers will always be ...
0
votes
0answers
8 views

Calculating reliability for scales with cross-loaded items

I am reexamining the second-order factor structure of a well-known multidimensional scale (8 first order factors and 3 second order factors). Using CFA in AMOS I have examined a number of models (I ...
0
votes
0answers
12 views

Negative Cronbach's Alpha

I run the Cronbach's Alpha based on the result of the Exploratory Factor Analysis, and 5 factors found. 1st is positive with Alpha, 2nd is low with Alpha, 3rd, 4th and 5th are having the negative ...
0
votes
0answers
25 views

Lower Cronbach's alpha when adding in reverse coded items

I have been working on some scale development and have administered a series of questions in different phases. The last time I administered the questions I had good structural validity and reliability ...
1
vote
0answers
15 views

Does it make sense to use Cronbach's $\alpha$ to check if questions do NOT pertain to the same construct?

I have three questions in a survey that are supposed to measure different constructs. However, I have my doubts if the respondents will not confuse them. So, I know that Cronbach's $\alpha$ is used to ...
0
votes
0answers
26 views

Can Cohen's kappa be used for raters with different but related targets?

There are two binary raters: Rater 1 is certified to assess the ability of people to drive a taxi. Rater 2 is certified to assess the ability of people to drive a lorry. Rater 1 and 2 have both ...
0
votes
1answer
36 views

EFA SPSS and Cronbach's Alpha

I have done the EFA in SPSS for my questionnaires and found 4 factors. Now should I use this EFA result to compute the Cronbach's Alpha, and if so then how to do that? Or Cronbach's Alpha is done ...
0
votes
1answer
67 views

Average inter-item correlations as internal consistency indicator

In literature, the average inter-item correlations in the range of 0.15 to 0.50 is considered as an indicator for an acceptable level of consistency. Why do the high average inter-item correlations ...
1
vote
0answers
37 views

How can measure be unreliable but valid?

Some previous research found evidence for the predictive validity of a measure. I've done some work and found the measure to be unreliable. Since reliability provides an upper limit for predictive ...
0
votes
1answer
28 views

Computing Inter-Rater Reliability - Different subjects are rated by different subsets of coders

I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each ...
1
vote
0answers
28 views

Can I draw a single ROC curve for a test method applied onto a group of people with different levels (i.e., light, mild,sever) of sickness

Can I draw a single ROC curve for a test method applied onto a group of people of different levels of sickness severity (i.e., light, mild, severe)? Or I need to draw three different ROC curves, one ...
0
votes
1answer
40 views

Why is Cohen's kappa low despite high observed agreement?

I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can ...