A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki).

learn more… | top users | synonyms

4
votes
2answers
45 views

Accuracy vs. area under the ROC curve

I constructed an ROC curve for a diagnostic system. The area under the curve was then non-parametrically estimated to be AUC = 0.89. When I tried to calculate the accuracy at the optimum threshold ...
1
vote
0answers
4 views

How to obtain statistical information from a Likert scale with a sample size smaller than the number of questions?

Research and sample properties I'm measuring the degree of a certain concept related to marketing and sustainable development throughout an entire industry. The theoretical model is depicted next: ...
0
votes
0answers
17 views

Reliability & Validity of modified /adapted scale

I have modified a scale applicable for employees in organizations to athletes in sports. Except changing a few words applicable for the target population no other changes are brought. The four ...
0
votes
0answers
9 views

Would you say this is a good use of Mcdonald Omega Coefficient

This forum was extremely helpful in nudging me towards considering an alternative to Cronbach's Alpha. I found the critique on the alpha riveting. While I accept that the alpha is an "industry ...
0
votes
0answers
9 views

Job motivation survey validity and reliability

A survey was done to measure job motivation of the workers in a community service provider. There were 13 items in the questionnaire and each item was on 10-point likert scale. A score will be ...
1
vote
0answers
11 views

Statistical comparison between two stochastic algorithms

I am working on a mechanics problem with variability in material properties. For that I need to analyze the efficiency of two methods for same accuracy (confidence level I guess?), the first one being ...
0
votes
0answers
13 views

Inter-rater reliability for time series annotations

I'm having trouble finding measures for inter-rater reliability in case of annotations that are time series. In this case, annotators rated a video of subjects for a particular variable, using a ...
1
vote
0answers
8 views

Factor analysis - Low and High conditions

So I'm conducting a factor analysis and also want to test the reliability of the scale. I have 2 conditions (between subjects) : low involvement and high involvement, both composed by the same items. ...
1
vote
0answers
5 views

Minimal Detectable Change for population surveys

I am wondering if it would be possible to calculate the Minimal Detectable Change when examining 2 population surveys? I know how to calculate the MDC in a test-retest situation in which every ...
0
votes
0answers
24 views

consistency of a task with varying number of items

How does one measure the internal consistency of a task that is finished at different times for each participant? Most psychometric tests have instructions that inform you that you must finish the ...
0
votes
1answer
23 views

What is the proper inter-rater reliability analysis for continuous count data?

I'm analyzing a study where participants needed to list as many plausible solutions to a given question. There are two raters who are counting the number of plausible solutions, what is the best way ...
0
votes
0answers
29 views

Cronbach's Alpha - reliability

In psychometric tests, we sometimes are instructed to end the test after the child has failed a determined number of items. Thus we end up with children who answered 30 questions and some who answered ...
0
votes
1answer
8 views

internal reliability scores below .70

When one is assessing the reliability of one instrument and the value is under .70 and you then eliminate the item/items that are mentioned that if eliminated would increase the reliability score, ...
0
votes
1answer
38 views

internal reliability

I have been searching for information on how to conduct internal reliability assessments but have been unsuccessful. Is there a book or website that can inform me on how to conduct these analysis? I ...
1
vote
0answers
31 views

Dealing with survival analysis data - reliability

I am totally new on Survival Analysis and my data is tricking me. I have, in excel, a list of failure's records informing date and hour of ocurrence, what failed/broke on that equipment and the cause ...
0
votes
0answers
25 views

Is my survey invalid now?

I'm running an online survey for my thesis. In order for the survey to be valid, I needed a minimum of 375 participants. And I have achieved this number. However, upon closer inspection I have ...
0
votes
1answer
45 views

Given a 1-Sided Lower Confidence Bound value, How to determine target failure rate

$R(t)$ = reliability = 1 - unreliability (probability that item is still operational for a given time t) unreliability = Cumulative Distribution Function (CDF) = (...
0
votes
1answer
38 views

What type of study?

I calculated inter-rater reliability for a medical test. We applied the test on both healthy people and people with a specific disease and then two raters interpreted the test. Can I call this study a ...
0
votes
0answers
15 views

Ordinal Alpha Reliability: Is Re-Ranking Appropriate?

I'm a researcher currently based in New Delhi where I'm helping adapt and validate a psychometric questionnaire. For this I need help with fixing the error in calculating ordinal alpha in R using the ...
2
votes
1answer
25 views

General validity/reliability of questionnaires

I have a general question concerning the reliability/validity of questionnaires. I'm working in Human Resources and we plan to conduct a workplace engagement survey. I attended a conference and a ...
0
votes
1answer
17 views

Inter-rater reliability simultaneously on multiple binary data

I have drug prescription records on 5 different drugs (A,B,C,D,E) on 2 different time points over a bunch of patients. 1/0 stands for use or not-use the specific drug, respectively. My questions are,...
1
vote
0answers
31 views

NVivo IRR Test of Content Analysis Data: How do I interpret each row of Kappas?

I am currently working on a research project that involves content analysis of secondary published data (corporate sustainability reports) by U.S. companies. In order to capture the data, I and two ...
1
vote
1answer
32 views

what am i supposed to do if realiability test not up to 0.7?

hye. i have gathered 147 questionnaires with 39 variables the requirement for my research department need me to make the reliability test up to 0.7 or else it consider not reliable. i have tried ...
1
vote
1answer
28 views

Intuition about f1 score

Let's say I have a data set where half of the data points are labelled as positive and half are labelled as negative. My task is to create a classifier which recognises when a sample from the dataset ...
0
votes
0answers
51 views

Estimation of the probability of the winner's curse

The increase in false-positive results arises primarily from an overestimation of effect sizes, leading to upward-biased effect sizes that often cannot be reproduced in follow-up studies (‘the winner'...
0
votes
0answers
16 views

Reliability of measurement scale of non-normal variables

I have 18 variables (measured in a Likert-scale) which should according to previous studies measure the same thing, let's say intelligence. I want to calculate the reliability of the sum variable of ...
0
votes
0answers
7 views

Formula to convert test-retest reliability into SD of difference

Wanting to work out the statistical power of a repeated-measures t.test. This requires an estimate of the SD of differences within pairs (the correlation between measures for a given person is what ...
0
votes
0answers
35 views

Principal Component Analysis: Scale reliability and validity

I'm using a popular psychometric scale, the Domain Specific Risk Taking (DoSpeRT) scale, that contains 30 items that measure an individual's risk taking capacity across five different dimensions: ...
0
votes
1answer
46 views

What to do with negative ICC values? Adjust the test or interpret it differently?

For my graduation thesis I am doing a study for the test-retest reliability of the tendon thickness of a particular muscle. So this study contains one rater and 70 subjects who have been tested at two ...
1
vote
1answer
17 views

Test Retest of highly skewed dichotomous variables

Does anyone have recommendations on how to evaluate test retest reliability with 95% CI (individuals took the same survey 1 week apart) on data that is dichotomous (yes = 1; no = 0) where the data is ...
0
votes
0answers
12 views

Access validity and reliability for hypotheses testing in prediction model

In my bachelor thesis I'm investigating a transaction data set of a retailer. I'm trying to find out indicators that predict a certain outcome, which is why I proposed several hypotheses for the ...
0
votes
0answers
17 views

1-Sided Lower Confidence Level Concept for Reliability (1-F(T))

For reliability (R) engineering discipline, reliability of an item is the conditional probability that the item will be functional within the mission duration at a given confidence level. so ...
0
votes
0answers
6 views

Is there a common recommendation on product lifetime declarations in terms of Reliability function?

I mean: if I read on the label that a specific product (car, household appliance, whatever...) is declared to have a lifetime of N years, does it actually mean that the Reliability function R(t), ...
0
votes
0answers
76 views

Cronbach Alpha Assumptions

I'm currently learning ho to use Cronbach Alpha in R. I have a scale with 7 items and 63 respondents. The aim is just to get a practical understanding of what Cronbach Alpha is. There are some nulls ...
0
votes
0answers
36 views

Calculate Cronbach's alpha for three observers

I have a dataset which contains 100 observations, and I asked three expert to describe each observation by one of the letters {A,B,C}. The following three tables shows a sample of the dataset with the ...
1
vote
1answer
46 views

Clear explanation of Cronbach's Alpha

I have difficulty in understanding Cronbach's Alpha's formula. I searched for google and StackExchange able to understand it. Even there are some topics trying to explain it intuitively, I could not ...
0
votes
0answers
13 views

Compute interrater reliability having different subsets of students rated by different subsets of raters

I would like to compute inter-rater reliability for a set of continuous items, having different subsets students assessed by different subsets of raters (please, see the picture). I would like to ...
2
votes
1answer
34 views

Cronbach's alpha vs. Average Item-Sum correlation?

I have heard that Cronbach's alpha is used to measure reliability of a test (to check if a set of items/question are measuring the same construct). How does the Cronbach's alpha compare with the ...
1
vote
1answer
14 views

Randomization scheme for rating images and assessing inter-rater reliability

Apologies for the basic question, but I'm new to the area and I'm working on a research project on rating images, and assessing the reliability of these ratings, with the following structure: There ...
0
votes
0answers
12 views

How to estimate the reliability of an exhaustive list of data?

I have a purported exhaustive list of beneficiaries under a social security scheme. My question is how large must be the sample of this population must be tested to check for fraud etc to derive an ...
0
votes
1answer
23 views

Intrarater reliability test for a single subject?

I have repeatedly measured the volume of the brain of 1 subject in an MRI study. Measurements were acquired by the same rater (myself) on three separate occasions resulting in three measurements of ...
1
vote
0answers
26 views

Effect of reliability on t-test

I have some problem with investigating the effect of reliability on the t-test. Basically it is assumed that lower reliability leads to a broader confidence interval or higher SD, which has a ...
0
votes
1answer
63 views

How can I calculate percent agreement and kappa for each category?

My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges ...
1
vote
0answers
38 views

Goodness of Fit and reliability of coefficients in Logistic Regression

I have logistic regression model with 9 independent variables. Two of them are numerical, the others are binary. Three of my coefficients are significant at 0.001, two of them are significant at 0.05. ...
1
vote
1answer
20 views

What to report for reliability analysis?

After completing data collection on a reliability analysis where multiple examiners sampled each subject three times, which value should I report for descriptive statistics? If intra-rater reliability ...
0
votes
1answer
39 views

Why is my Cronbach's $\alpha$ negative [duplicate]

I'm just learning about Cronbach's $\alpha$. I have recently tried to carry out this test. Having 80 completed questionnaires with 15 questions (variables), the questionnaire used the Likert scale 1-5 ...
0
votes
1answer
122 views

Inter-rater reliability measure with multiple categories per item [duplicate]

I have multiple raters providing categorical ratings. What makes this slightly different from other posted questions is that each rater can provide three, unique categorical ratings. Each rater had ...
0
votes
0answers
21 views

Effect of reliability on sample size

I asked a similar question here: Meta-analysis using experimental reliability data However as I did not get any feedback (probably for a good reason, which I would like to understand) I ask the ...
1
vote
0answers
21 views

Combining the standard errors of Cohen's Kappa computed for multiple rater pairs

In a study on intra-rater agreement with 8 raters, I have computed Cohen's Kappa for each of the raters and I estimated the standard error of each of the kappa values using the Jackknife method. To ...
0
votes
1answer
56 views

Normalization of values

How do I normalize a set of data? Let's say we are looking at measurements of diameter of the heart in women and men. Men will have bigger hearts, so error in measuring by the observers will always be ...