Questions tagged [validity]
An instrument is valid if it measures what it purports to measure. This is different from reliability, which means that an instrument measures (whatever it measures) in a way that is repeatable. DO NOT use this tag for `validation` model purposes.
136 questions
2
votes
1
answer
41
views
Randomization validity based on a modulus function
I want to run an AB test on a platform that only supports assigning variants based on address_id. Each user can have multiple address_ids in our platform i.e each user_id can be mapped to multiple ...
- 723
0
votes
1
answer
625
views
Validity of ARDL regression when there is I(2) variable
I am trying to model relationship between Y(t) and Y(t-1), X(t), X(t-1) and Z(t) using ARDL model. Most cases these time-series are I(0) or I(1). However sometimes I encounter I(2) or higher order.
...
CommunityBot
- 1
1
vote
1
answer
2k
views
Doing CFA on a known theoretical model, but having problems with convergent and discriminant validity
I did a survey based on the UTAUT model / theory with standard questions used. The N was 150. I am trying to validate the model for my survey with structural equation modeling (via the AMOS software). ...
CommunityBot
- 1
1
vote
0
answers
18
views
Clustering and samples weighting (using GEE)
I am currently analyzing the social factors associated with the prevalence of infectious diseases in European countries using GEE models. My data is based on patients covered by a specific health ...
- 11
0
votes
0
answers
14
views
calculation of the C-index clustering for manual [duplicate]
Can anyone give me an example of working on the C-index clustering validity test, but calculating manually??
- 1
5
votes
1
answer
46
views
Validating binary prediction model
Suppose we have a model that predicts for binary event $e$ ($0$ or $1$) with a single output $p$ (the expected probability $e$ occurs).
If we are able to compare $p$ with the true value of $e$ ($0$ or ...
- 67.1k
0
votes
0
answers
8
views
Can you build up statistical validity with multiple month's worth of the same survey questions?
The company I work for conduct consulting where we analyse company survey responses for statistical validity against the general company population.
To prevent survey fatigue, the company sends out ...
- 1
1
vote
0
answers
31
views
Gender Differences in Subscales [closed]
I need to demonstrate discriminant validity for two subscales (subscales have 6 and 7 items) that should differ between males and females. There is a theoretical basis that there should be a ...
- 128k
2
votes
2
answers
57
views
In a regression, is an independent variable still useful if it is the same for each subject between observations?
I was not able to find an answer to this question, as I feel it is rather specific and awkward to ask. But, here goes. This is a simplified version of my data:
As you can see, I have various subjects ...
- 3,427
0
votes
0
answers
12
views
Validity, reliability and repeatability
I want to validate an anemometer in a wind tunnel VS gold standard.
Thinking in validity, reliability and repeatability.
The dependent variable is wind speed. I can test 3 different units of the same ...
- 91
1
vote
3
answers
1k
views
What is meant by practical significance and statistical significance? How do we distinguish these concepts? [duplicate]
I am unable to understand content reported in :
Practical vs Statistical significance
Statistical significance-testing implies whether a sample-statistic matches with population estimate of effect-...
- 18.4k
8
votes
2
answers
6k
views
How high is too high with Cronbach's alpha?
Someone in my organization just sent me a scale they developed (modified items from five separate scales used in published research) to measure an employee's readiness for change. They tested the ...
- 2,939
0
votes
1
answer
42
views
Validation Checklist Passing Score
I have a mixed-method thesis ongoing. I'm currently on the validation stage of my survey and interview questions. I have adopted a validation checklist from the internet composing of 15 items with a 4-...
- 128k
1
vote
0
answers
453
views
How to test relative validity statistically?
I am trying to understand a) what relative validity is, and b) how to test it statistically. So far I have gathered, from sources such as this that relative validity is a larger reliability in ...
- 82.8k
3
votes
3
answers
172
views
What is the (logical) relationship between an measurement instrument's validity and responsiveness?
The three major criteria for the quality of a measurement instrument are validity, reliability, and responsiveness. I understand validity and reliability, but I have trouble understanding the need to ...
- 128k
5
votes
0
answers
101
views
How can statistics be used to avoid "Lending False Credibility To Decisions We've Already Made" [closed]
In light of this article Data Science Has Become About Lending False Credibility To Decisions We've Already Made published in Forbes, I would appreciate input from the statistical and data science ...
- 57
0
votes
0
answers
10
views
Validating a new definition for detection
I am having some trouble understanding what approach I would use to validate a new set of definitions versus the "gold standard" to detect disease positivity. For example, if I wanted to say ...
- 75
3
votes
0
answers
76
views
Difference between kappa statistics
I am currently developing an instrument and have some questions on the use of kappa statistics for interrater agreement used for content validity. I have gone through previous literature and I still ...
- 82.8k
0
votes
0
answers
22
views
Estimate the threohold of reject region by Bootstrap
I am studying the Bootstrap method and encountered a question about how to estimate the reject region. Suppose we have observed a sample $\{(X_i, Y_i)\}_{i=1}^n$, I aim to investigate the linear ...
- 43
1
vote
1
answer
21
views
Approach of a Golden Standard Measurement with questionnaire data
I am a bit lost on the correct approach to this Statistical problem. Let me explain the context of this problem: I have a cohort population with longitudinal measurements during a period of several ...
- 19k
0
votes
1
answer
158
views
Is it appropriate to use log-rank test in a study having multiple cases with expected values less than 5?
My goal is to show independency of factors related to survival of a small size sample (26 patients) for a period of time, this sample was divided in 2 groups (16 vs 10 patients), the events were ...
- 101k
3
votes
1
answer
333
views
Can I check the reliability of my test after collecting my data?
I have performed a study where I tested 2 groups, before and after two different types of manipulations. The main outcome of my study were the group differences at time point 2.
Now someone asks what ...
- 28.3k
1
vote
0
answers
11
views
Gender Differences in Validity
I need to demonstrate discriminant validity for two subscales (subscales have 6 and 7 items) that should differ between males and females. There is a theoretical basis that there should be a ...
- 11
4
votes
1
answer
39
views
The validity of using paired sample t-test to compare results from two different test methods
My research involved two types of contextualized tests to examine two types of knowledge of collocations. The first test is a multiple-choice (with 3 options) and it was used to examine receptive ...
- 18.4k
1
vote
1
answer
27
views
What is the difference between validity testing and actual results?
Researches usually present the results of validity tests conducted before the actual results - these include descriptive statistics with Mean and SD and also inter-construct correlations.
Then there ...
- 19k
1
vote
1
answer
86
views
Reliable methods to validate clustering of text phrases?
Question is in the title. I have clustered the word embeddings of text phrases, and now want to try and check whether the resulting clusters are coherent enough. I have tried methods that are ...
- 11.2k
1
vote
0
answers
93
views
Why is internal consistency a measure of reliability?
Why is 'internal consistency' a measure of reliability? It looks like a measure of validity instead.
Definitions:
"Internal consistency reflects the extent to which items within an instrument ...
- 485
1
vote
0
answers
72
views
How to meta-analyze incremental validity
Suppose we have two instruments (A and B) and we want to meta-analyze whether instrument B has incremental validity over instrument A in predicting an outcome Y. What effect size would be most ...
- 65
0
votes
0
answers
41
views
Instrument validation: Proportional-odds ordinal regression?
I am looking for advice on the best statistical tests to use to accomplish my goal. I am making a new instrument to measure a variable and want to compare it to the current (gold standard) instrument. ...
- 1
0
votes
0
answers
100
views
colinearity in stepwise regression
I would be very grateful if you can be so kind to clarify what may be wrong in the following steps and interpretation of the validity conditions of the stepwise regression.
I am afraid there is a ...
- 82.8k
1
vote
1
answer
2k
views
Placebo test for the validity of difference-in-differences analysis results for limited data period
I am doing an analysis of a strategy implemented by a four-year-old corporate company and this implementation was given in different cities which all of its employees were received the treatment or ...
CommunityBot
- 1
1
vote
2
answers
72
views
I want to test for a significant difference in the result (t-value or effect size) of two paired t-tests
I am testing the quality of a measurement construct, where I propose that Measure A is better (i.e. more responsive to situational changes) than Measure B is.
I conduct a study in which I measure the ...
- 3
0
votes
1
answer
61
views
Validity of multilevel modeling to include results for multiple psychometric tests with subscales #statsnube
I have survey data for approx n = 1650 for multiple psychometric tests (all participants have completed all the tests), and about 12 outcome variables (from a PCA of a 52-question survey, e.g. ...
- 11
1
vote
1
answer
105
views
Instrument validity: does a positive and significant coefficient on Z in a regression of Y on X and Z pose a problem?
I have an initial regression of Y on X and Z. Both of my coefficients on X and Z are non-zero and strongly statistically significant. X and Z are correlated but I am told collinearity shouldn't be an ...
- 858
1
vote
1
answer
331
views
Confirmatory Factor Analysis : validity for a just-identified model
I have developed a 3-item questionnaire, and I would like to test the validity of this questionnaire. I wanted to test the validity using confirmatory factor analysis, but since my model is just-...
- 221
0
votes
0
answers
543
views
Testing validity of an instrumental variable with exact identification
I am conducting an instrumental variable regression. I have one endogenous variable and one instrument? How can I test if the instrument is valid? What I have done for now is run the ivreg command and ...
- 111
1
vote
1
answer
159
views
Discussing validity of tests performed after a cluster analysis
I'm new to datascience (from a medical/medical science background).
My supervisor (social sciences background) asked me to assist in rewriting a paper where we do a cluster analysis for a ...
5
votes
1
answer
17k
views
How are Cronbach's Alpha and Average Variance Extracted different in their interpretation?
Background: I do survey research on Information Systems. I have a latent construct that I measure with six survey items and reflective measurement model. Each survey item has a seven point Likert ...
- 69.5k
2
votes
0
answers
401
views
2SLS approach using lagged dependent variable as an instrument
I was wondering if the lagged dependent variable can be a valid instrument or not.
In my data, policy intervention (x) is made depending on the outcome (y) in the past. (not depending on the effect (b)...
- 61
1
vote
0
answers
132
views
Propensity score matching at the group level and models at the subject level
I am running a study that uses a combination of propensity score matching and mixed-effects models. I use propensity score matching to locate some groups (half of them represent control, and another ...
- 87
1
vote
1
answer
73
views
Can reliability and validity assessments be considered as required constituent of the "Measurement part" in Structural Equation Modeling?
I wonder if any type of reliability and / or validity analysis can be considered as an element of the Measurement part in SEM. For example, when computing Cronbach's alpha or the AVEs of an instrument,...
- 171
2
votes
0
answers
28
views
2stage (instrumental variable) analysis: is reverse causality in the first stage a problem? [duplicate]
For my research I am looking into the relationship between an outcome (Y) and a predictor (X) as follows:
$$Y = X + e$$
where $e$ is the error term.
Because there might be reverse causality I am ...
- 822
3
votes
1
answer
737
views
How would you validate a new test without a gold standard
I am measuring a variable (continuous, unknown distribution) on a new assessment tool. There is no gold standard to compare to. I can take 2 measures per participant only. There is no intervention ...
- 13.3k
3
votes
1
answer
503
views
How to test the inter-rater reliability of a nominal coding process that generates ordinal data?
A researcher studying hunger on a tropical island asks respondents, "What did you eat for lunch yesterday?" and the possible choices are "(4) Nothing; (3) a banana; (2) an unripe coconut; (1) a ripe ...
CommunityBot
- 1
3
votes
2
answers
1k
views
Does changing the point range of items of a construct measured through likert scale affect in any way validity and reliability of the construct?
I am using a pre-existing universal tested scale to measure a specific construct. The construct is measured through 5 items on a 5 points likert scale. However, I would probably need to change the ...
- 59.5k
2
votes
0
answers
1k
views
Comparison between two measurement devices
I have two measurement devices accelerometer A and B. The measurement is the step time of a person walking in a corridor. A is the gold standard. I want to check the validity and reliability of B. how ...
- 54.3k
1
vote
1
answer
6k
views
item-total correlation vs. inter-item correlation
I am assesing the reliability and validity of an 16-item multuiple choice test supposed to test language skills. Cronbach's apha is "questionable" with .65. I am having troubles interpreting the ...
CommunityBot
- 1
4
votes
1
answer
3k
views
What's an intuitive way to explain the different types of validity?
Specifically I'm thinking of a simplified division whereby validity is divided into:
Construct validity
1a. Convergent validity
1b. Discriminant validity
Criterion related validity
2a. Predictive ...
CommunityBot
- 1
0
votes
2
answers
156
views
Is this an instance of convergent evidence?
In psychometrics, convergent validity is a source of proof for construct validity.
It is said that it is the degree to which test scores are correlated with test scores of relevant constructs or the ...
- 836
0
votes
0
answers
114
views
Validity with repeated measures
I have some data that was collected with 2 different devices on a sample of 50 subjects. It was also collected for 50 days in a row.
One of the devices is considered "the gold standard" ...
- 1