Questions tagged [validity]
An instrument is valid if it measures what it purports to measure. This is different from reliability, which means that an instrument measures (whatever it measures) in a way that is repeatable. DO NOT use this tag for `validation` model purposes.
136 questions
2
votes
1
answer
41
views
Randomization validity based on a modulus function
I want to run an AB test on a platform that only supports assigning variants based on address_id. Each user can have multiple address_ids in our platform i.e each user_id can be mapped to multiple ...
- 21
1
vote
0
answers
18
views
Clustering and samples weighting (using GEE)
I am currently analyzing the social factors associated with the prevalence of infectious diseases in European countries using GEE models. My data is based on patients covered by a specific health ...
- 11
0
votes
0
answers
14
views
calculation of the C-index clustering for manual [duplicate]
Can anyone give me an example of working on the C-index clustering validity test, but calculating manually??
- 1
5
votes
1
answer
46
views
Validating binary prediction model
Suppose we have a model that predicts for binary event $e$ ($0$ or $1$) with a single output $p$ (the expected probability $e$ occurs).
If we are able to compare $p$ with the true value of $e$ ($0$ or ...
- 161
0
votes
0
answers
8
views
Can you build up statistical validity with multiple month's worth of the same survey questions?
The company I work for conduct consulting where we analyse company survey responses for statistical validity against the general company population.
To prevent survey fatigue, the company sends out ...
- 1
2
votes
2
answers
57
views
In a regression, is an independent variable still useful if it is the same for each subject between observations?
I was not able to find an answer to this question, as I feel it is rather specific and awkward to ask. But, here goes. This is a simplified version of my data:
As you can see, I have various subjects ...
- 21
0
votes
0
answers
12
views
Validity, reliability and repeatability
I want to validate an anemometer in a wind tunnel VS gold standard.
Thinking in validity, reliability and repeatability.
The dependent variable is wind speed. I can test 3 different units of the same ...
- 91
0
votes
1
answer
42
views
Validation Checklist Passing Score
I have a mixed-method thesis ongoing. I'm currently on the validation stage of my survey and interview questions. I have adopted a validation checklist from the internet composing of 15 items with a 4-...
- 41
3
votes
3
answers
172
views
What is the (logical) relationship between an measurement instrument's validity and responsiveness?
The three major criteria for the quality of a measurement instrument are validity, reliability, and responsiveness. I understand validity and reliability, but I have trouble understanding the need to ...
- 131
1
vote
3
answers
1k
views
What is meant by practical significance and statistical significance? How do we distinguish these concepts? [duplicate]
I am unable to understand content reported in :
Practical vs Statistical significance
Statistical significance-testing implies whether a sample-statistic matches with population estimate of effect-...
- 161
0
votes
0
answers
10
views
Validating a new definition for detection
I am having some trouble understanding what approach I would use to validate a new set of definitions versus the "gold standard" to detect disease positivity. For example, if I wanted to say ...
- 75
3
votes
0
answers
76
views
Difference between kappa statistics
I am currently developing an instrument and have some questions on the use of kappa statistics for interrater agreement used for content validity. I have gone through previous literature and I still ...
- 31
0
votes
0
answers
22
views
Estimate the threohold of reject region by Bootstrap
I am studying the Bootstrap method and encountered a question about how to estimate the reject region. Suppose we have observed a sample $\{(X_i, Y_i)\}_{i=1}^n$, I aim to investigate the linear ...
- 43
1
vote
1
answer
21
views
Approach of a Golden Standard Measurement with questionnaire data
I am a bit lost on the correct approach to this Statistical problem. Let me explain the context of this problem: I have a cohort population with longitudinal measurements during a period of several ...
- 11
0
votes
1
answer
158
views
Is it appropriate to use log-rank test in a study having multiple cases with expected values less than 5?
My goal is to show independency of factors related to survival of a small size sample (26 patients) for a period of time, this sample was divided in 2 groups (16 vs 10 patients), the events were ...
- 37
3
votes
1
answer
333
views
Can I check the reliability of my test after collecting my data?
I have performed a study where I tested 2 groups, before and after two different types of manipulations. The main outcome of my study were the group differences at time point 2.
Now someone asks what ...
- 309
1
vote
0
answers
11
views
Gender Differences in Validity
I need to demonstrate discriminant validity for two subscales (subscales have 6 and 7 items) that should differ between males and females. There is a theoretical basis that there should be a ...
- 11
4
votes
1
answer
39
views
The validity of using paired sample t-test to compare results from two different test methods
My research involved two types of contextualized tests to examine two types of knowledge of collocations. The first test is a multiple-choice (with 3 options) and it was used to examine receptive ...
- 41
1
vote
0
answers
31
views
Gender Differences in Subscales [closed]
I need to demonstrate discriminant validity for two subscales (subscales have 6 and 7 items) that should differ between males and females. There is a theoretical basis that there should be a ...
- 11
1
vote
1
answer
27
views
What is the difference between validity testing and actual results?
Researches usually present the results of validity tests conducted before the actual results - these include descriptive statistics with Mean and SD and also inter-construct correlations.
Then there ...
- 31
1
vote
1
answer
86
views
Reliable methods to validate clustering of text phrases?
Question is in the title. I have clustered the word embeddings of text phrases, and now want to try and check whether the resulting clusters are coherent enough. I have tried methods that are ...
- 11
1
vote
0
answers
93
views
Why is internal consistency a measure of reliability?
Why is 'internal consistency' a measure of reliability? It looks like a measure of validity instead.
Definitions:
"Internal consistency reflects the extent to which items within an instrument ...
- 485
1
vote
0
answers
72
views
How to meta-analyze incremental validity
Suppose we have two instruments (A and B) and we want to meta-analyze whether instrument B has incremental validity over instrument A in predicting an outcome Y. What effect size would be most ...
- 65
0
votes
0
answers
41
views
Instrument validation: Proportional-odds ordinal regression?
I am looking for advice on the best statistical tests to use to accomplish my goal. I am making a new instrument to measure a variable and want to compare it to the current (gold standard) instrument. ...
- 1
0
votes
0
answers
100
views
colinearity in stepwise regression
I would be very grateful if you can be so kind to clarify what may be wrong in the following steps and interpretation of the validity conditions of the stepwise regression.
I am afraid there is a ...
- 1
1
vote
2
answers
72
views
I want to test for a significant difference in the result (t-value or effect size) of two paired t-tests
I am testing the quality of a measurement construct, where I propose that Measure A is better (i.e. more responsive to situational changes) than Measure B is.
I conduct a study in which I measure the ...
- 11
0
votes
1
answer
61
views
Validity of multilevel modeling to include results for multiple psychometric tests with subscales #statsnube
I have survey data for approx n = 1650 for multiple psychometric tests (all participants have completed all the tests), and about 12 outcome variables (from a PCA of a 52-question survey, e.g. ...
- 11
1
vote
1
answer
105
views
Instrument validity: does a positive and significant coefficient on Z in a regression of Y on X and Z pose a problem?
I have an initial regression of Y on X and Z. Both of my coefficients on X and Z are non-zero and strongly statistically significant. X and Z are correlated but I am told collinearity shouldn't be an ...
- 11
1
vote
1
answer
330
views
Confirmatory Factor Analysis : validity for a just-identified model
I have developed a 3-item questionnaire, and I would like to test the validity of this questionnaire. I wanted to test the validity using confirmatory factor analysis, but since my model is just-...
- 99
0
votes
0
answers
543
views
Testing validity of an instrumental variable with exact identification
I am conducting an instrumental variable regression. I have one endogenous variable and one instrument? How can I test if the instrument is valid? What I have done for now is run the ivreg command and ...
- 111
1
vote
1
answer
159
views
Discussing validity of tests performed after a cluster analysis
I'm new to datascience (from a medical/medical science background).
My supervisor (social sciences background) asked me to assist in rewriting a paper where we do a cluster analysis for a ...
- 11
2
votes
0
answers
401
views
2SLS approach using lagged dependent variable as an instrument
I was wondering if the lagged dependent variable can be a valid instrument or not.
In my data, policy intervention (x) is made depending on the outcome (y) in the past. (not depending on the effect (b)...
- 61
1
vote
0
answers
132
views
Propensity score matching at the group level and models at the subject level
I am running a study that uses a combination of propensity score matching and mixed-effects models. I use propensity score matching to locate some groups (half of them represent control, and another ...
- 87
2
votes
0
answers
28
views
2stage (instrumental variable) analysis: is reverse causality in the first stage a problem? [duplicate]
For my research I am looking into the relationship between an outcome (Y) and a predictor (X) as follows:
$$Y = X + e$$
where $e$ is the error term.
Because there might be reverse causality I am ...
- 73
0
votes
2
answers
156
views
Is this an instance of convergent evidence?
In psychometrics, convergent validity is a source of proof for construct validity.
It is said that it is the degree to which test scores are correlated with test scores of relevant constructs or the ...
- 1
0
votes
0
answers
113
views
Validity with repeated measures
I have some data that was collected with 2 different devices on a sample of 50 subjects. It was also collected for 50 days in a row.
One of the devices is considered "the gold standard" ...
- 1
0
votes
0
answers
29
views
Algorithm to calculate validity of document?
I have a dataset of text-documents, which are all considered valid. Also I have a set of manually defined features that are required to be in the document for it to be considered valid. The features ...
- 251
1
vote
1
answer
2k
views
Placebo test for the validity of difference-in-differences analysis results for limited data period
I am doing an analysis of a strategy implemented by a four-year-old corporate company and this implementation was given in different cities which all of its employees were received the treatment or ...
- 11
1
vote
1
answer
73
views
Can reliability and validity assessments be considered as required constituent of the "Measurement part" in Structural Equation Modeling?
I wonder if any type of reliability and / or validity analysis can be considered as an element of the Measurement part in SEM. For example, when computing Cronbach's alpha or the AVEs of an instrument,...
- 171
1
vote
0
answers
11
views
Measuring Validity and reliability
Say you have an independent train ticket system where automatic refunds would be issued in case of a delayed arrival. To determine the arrival time of the train at a certain station you collect data ...
- 111
1
vote
1
answer
246
views
Correlation between questionnaire to test validity
I'm doing a validation of a questionnaire for my thesis and I should investigate the validity of my test with others one. To do so I've been asked to calculate the correlation between two scales and I'...
0
votes
0
answers
26
views
Questionnaire validity statistical significance
My colleagues and I, have developed a questionnaire of 116 questions, segmented in 7 constructs. Cronbach's a indices for each construct are above 70%, and it is content valid. However, when we apply ...
- 73
2
votes
0
answers
1k
views
Comparison between two measurement devices
I have two measurement devices accelerometer A and B. The measurement is the step time of a person walking in a corridor. A is the gold standard. I want to check the validity and reliability of B. how ...
- 21
0
votes
0
answers
28
views
Survey Question Validity
I have a survey that asks the user to report their debt from their undergraduate education and their debt from their graduate education. I intend to add these two averages to measure their average ...
- 159
0
votes
0
answers
28
views
Generalizability study of subscaled rating instrument
I have a school rating ($1$ thru $4$) instrument that consists of $9$ subscales (e.g., classroom instruction, school management etc.). Under each subscale, I have $6$ items on which rating occurs.
I ...
- 4,056
0
votes
1
answer
78
views
Is it always needed to do a validity test for variables?
Let's say I have one variable called "perceived image", which is measured with 6 items (questions) that I adapted from previous literature. If I have a Cronbach's alpha > 0.7 (for reliability), do I ...
- 7
3
votes
1
answer
299
views
Do I estimate factor loadings in a confirmatory factor anlysis (CFA) aimed at verifying an exploratory factor analysis (EFA)?
I decided to use a questionnaire published by another researcher (paper and supplementary here).
In the article they perform an EFA, find two factors, and report the resulting factor loadings (...
- 470
0
votes
0
answers
96
views
How to test nomological validity with lavaan?
in my understanding, I need to test nomological validity with SEM. But what exactly should I look at in e.g. summary(fit1, standardized=TRUE, fit.measures=TRUE)?
Code for reference:
...
- 569
0
votes
1
answer
625
views
Validity of ARDL regression when there is I(2) variable
I am trying to model relationship between Y(t) and Y(t-1), X(t), X(t-1) and Z(t) using ARDL model. Most cases these time-series are I(0) or I(1). However sometimes I encounter I(2) or higher order.
...
- 1
1
vote
1
answer
6k
views
item-total correlation vs. inter-item correlation
I am assesing the reliability and validity of an 16-item multuiple choice test supposed to test language skills. Cronbach's apha is "questionable" with .65. I am having troubles interpreting the ...
- 909