Questions tagged [validity]
An instrument is valid if it measures what it purports to measure. This is different from reliability, which means that an instrument measures (whatever it measures) in a way that is repeatable. DO NOT use this tag for `validation` model purposes.
121
questions
0
votes
0answers
5 views
What is the best way to handle longitudinal data where the follow-up question for the same outcome/construct has changed
I'm doing secondary analysis on a large longitudinal dataset (didn't collect myself) and have a series of questions related to movement/time-use behaviours (sleep, physical activity, sedentary ...
0
votes
0answers
10 views
Has a valid kernel to be positive in one dimension?
During my ML exercises, I have come across the kernel below:
$$ k(x,y) = sin(x)sin(y),\ x,y \in \mathbb{R}$$
The text is asking me if this is a valid kernel. My current knowledge would reply "yes&...
0
votes
1answer
12 views
Validity of multilevel modeling to include results for multiple psychometric tests with subscales #statsnube
I have survey data for approx n = 1650 for multiple psychometric tests (all participants have completed all the tests), and about 12 outcome variables (from a PCA of a 52-question survey, e.g. ...
2
votes
1answer
13 views
Instrument validity: does a positive and significant coefficient on Z in a regression of Y on X and Z pose a problem?
I have an initial regression of Y on X and Z. Both of my coefficients on X and Z are non-zero and strongly statistically significant. X and Z are correlated but I am told collinearity shouldn't be an ...
1
vote
1answer
22 views
Confirmatory Factor Analysis : validity for a just-identified model
I have developed a 3-item questionnaire, and I would like to test the validity of this questionnaire. I wanted to test the validity using confirmatory factor analysis, but since my model is just-...
0
votes
0answers
69 views
Calculating convergent and discriminant validity using R, via CFA
I've been trying to calculate convergent and discriminant validity using R, through CFA in lavaan package.
I'm trying to use the ...
0
votes
0answers
18 views
Testing validity of an instrumental variable with exact identification
I am conducting an instrumental variable regression. I have one endogenous variable and one instrument? How can I test if the instrument is valid? What I have done for now is run the ivreg command and ...
1
vote
1answer
32 views
Discussing validity of tests performed after a cluster analysis
I'm new to datascience (from a medical/medical science background).
My supervisor (social sciences background) asked me to assist in rewriting a paper where we do a cluster analysis for a ...
0
votes
0answers
22 views
Average variance extracted (AVE) with correlated error terms
after conducting a CFA I did some modifications to include correlated error terms (Error covariances) in my measurement models. Now I am wondering whether I need to apply a different formula für AVE / ...
1
vote
0answers
62 views
2SLS approach using lagged dependent variable as an instrument
I was wondering if the lagged dependent variable can be a valid instrument or not.
In my data, policy intervention (x) is made depending on the outcome (y) in the past. (not depending on the effect (b)...
1
vote
0answers
22 views
Propensity score matching at the group level and models at the subject level
I am running a study that uses a combination of propensity score matching and mixed-effects models. I use propensity score matching to locate some groups (half of them represent control, and another ...
2
votes
0answers
20 views
2stage (instrumental variable) analysis: is reverse causality in the first stage a problem? [duplicate]
For my research I am looking into the relationship between an outcome (Y) and a predictor (X) as follows:
$$Y = X + e$$
where $e$ is the error term.
Because there might be reverse causality I am ...
0
votes
0answers
20 views
Are cognitive interviews really a test of face validity?
When creating a questionnaire, cognitive interviews can be used to find out whether questions are easily understood, not upsetting, properly worded, not missing issues etc. However, is this really the ...
0
votes
0answers
41 views
How to calculate convergent and discriminant validity in SEM?
Using the average variance extract (AVE), the method is explained easily by doing some calculations on the factor loadings. I only found such procedure on CFA models, where we don't have an endogenous ...
0
votes
2answers
41 views
Is this an instance of convergent evidence?
In psychometrics, convergent validity is a source of proof for construct validity.
It is said that it is the degree to which test scores are correlated with test scores of relevant constructs or the ...
0
votes
0answers
26 views
Validity with repeated measures
I have some data that was collected with 2 different devices on a sample of 50 subjects. It was also collected for 50 days in a row.
One of the devices is considered "the gold standard" ...
0
votes
0answers
26 views
Algorithm to calculate validity of document?
I have a dataset of text-documents, which are all considered valid. Also I have a set of manually defined features that are required to be in the document for it to be considered valid. The features ...
0
votes
1answer
370 views
Placebo test for the validity of difference-in-differences analysis results for limited data period
I am doing an analysis of a strategy implemented by a four-year-old corporate company and this implementation was given in different cities which all of its employees were received the treatment or ...
0
votes
1answer
16 views
Can reliability and validity assessments be considered as required constituent of the "Measurement part" in Structural Equation Modeling?
I wonder if any type of reliability and / or validity analysis can be considered as an element of the Measurement part in SEM. For example, when computing Cronbach's alpha or the AVEs of an instrument,...
1
vote
0answers
9 views
Measuring Validity and reliability
Say you have an independent train ticket system where automatic refunds would be issued in case of a delayed arrival. To determine the arrival time of the train at a certain station you collect data ...
1
vote
1answer
30 views
Correlation between questionnaire to test validity
I'm doing a validation of a questionnaire for my thesis and I should investigate the validity of my test with others one. To do so I've been asked to calculate the correlation between two scales and I'...
0
votes
0answers
24 views
Questionnaire validity statistical significance
My colleagues and I, have developed a questionnaire of 116 questions, segmented in 7 constructs. Cronbach's a indices for each construct are above 70%, and it is content valid. However, when we apply ...
2
votes
0answers
239 views
Comparison between two measurement devices
I have two measurement devices accelerometer A and B. The measurement is the step time of a person walking in a corridor. A is the gold standard. I want to check the validity and reliability of B. how ...
0
votes
0answers
22 views
Survey Question Validity
I have a survey that asks the user to report their debt from their undergraduate education and their debt from their graduate education. I intend to add these two averages to measure their average ...
0
votes
0answers
25 views
Generalizability study of subscaled rating instrument
I have a school rating ($1$ thru $4$) instrument that consists of $9$ subscales (e.g., classroom instruction, school management etc.). Under each subscale, I have $6$ items on which rating occurs.
I ...
0
votes
1answer
27 views
Is it always needed to do a validity test for variables?
Let's say I have one variable called "perceived image", which is measured with 6 items (questions) that I adapted from previous literature. If I have a Cronbach's alpha > 0.7 (for reliability), do I ...
3
votes
1answer
118 views
Do I estimate factor loadings in a confirmatory factor anlysis (CFA) aimed at verifying an exploratory factor analysis (EFA)?
I decided to use a questionnaire published by another researcher (paper and supplementary here).
In the article they perform an EFA, find two factors, and report the resulting factor loadings (...
0
votes
0answers
42 views
How to test nomological validity with lavaan?
in my understanding, I need to test nomological validity with SEM. But what exactly should I look at in e.g. summary(fit1, standardized=TRUE, fit.measures=TRUE)?
Code for reference:
...
0
votes
1answer
62 views
Validity of ARDL regression when there is I(2) variable
I am trying to model relationship between Y(t) and Y(t-1), X(t), X(t-1) and Z(t) using ARDL model. Most cases these time-series are I(0) or I(1). However sometimes I encounter I(2) or higher order.
...
0
votes
1answer
2k views
item-total correlation vs. inter-item correlation
I am assesing the reliability and validity of an 16-item multuiple choice test supposed to test language skills. Cronbach's apha is "questionable" with .65. I am having troubles interpreting the ...
0
votes
0answers
191 views
Wilcoxon signed-rank test for paired and repeated measures data
I have a dataset with 45 participants, each with 1-7 datapoints/observations (126 total observations). Each of these observations represents a window of time in which two different methods attempt to ...
4
votes
0answers
62 views
How can statistics be used to avoid "Lending False Credibility To Decisions We've Already Made"
In light of this article Data Science Has Become About Lending False Credibility To Decisions We've Already Made published in Forbes, I would appreciate input from the statistical and data science ...
1
vote
1answer
16 views
Are there any studies of generalized error performance that don't assume data quality is constant with sample size?
As far as I know, much of the statistical and machine learning literature where modeling algorithms are compared for their generalization error performance as a function of sample size (think of ...
0
votes
0answers
13 views
factorial designed study within between
IV1 : alcohol use, IV2: breathing exercise (both of them within subject variable) IV3: gender (between subject variable). My research question is that: ''Does alcohol usage and breathing exercise's ...
0
votes
1answer
125 views
Calculating Confidence Interval with log-transformation
I fitted a linear regression with a 3rd order interaction and adjustment for confounding variable which looks like this:
$$
log(Y) = β_1S + β_2T + β_3M + β_4ST + β_5SM + β_6TM + β_7STM + β_XX
$$
...
0
votes
1answer
17 views
Can findings based on low values be considered valid?
Lets say i do have a TimeSeries with N > 300 but values like [0, 1, 2, 1, 2, 0, 2, ...] representing the visitors count of a website per day.
Since there are only few visitors and each of them can be ...
2
votes
1answer
117 views
What does "the hypotheses ... are verified as null" mean?
Context:
This article examines the criterion-related validity of the results obtained by the application of four different methods of assessment to the correction of a second-year exam of
...
2
votes
2answers
292 views
Reweighting features in PAM/K-means clustering
As stated in Hennig et al. 2016 Handbook of cluster analysis:
If for subject matter reasons some variables are more important than
others regardless of the within-variable variation, one could ...
9
votes
3answers
920 views
Does a statistic used as a measure becomes invalid after it's reported on? [closed]
Apologies if this the wrong place to ask. but i'm trying to explain to a director that his Statistics don't show the truth because he is demanding that the stats are what is reported on and staff are ...
0
votes
1answer
97 views
How to assess predictive validity of a variable on the outcome?
I am looking to assess the predictive validity of two measures (M1 and M2) on a binary outcome. After constructing my model, do I just calculate odds ratios and compare the odds ratios for the two ...
1
vote
0answers
55 views
Measuring emotion differentiation on a within-person level.
To measure emotion differentiation, which refers to the precision of an individual to accurately describe and distinguish between different emotional states, many studies have used a between-person ...
3
votes
2answers
253 views
Does changing the point range of items of a construct measured through likert scale affect in any way validity and reliability of the construct?
I am using a pre-existing universal tested scale to measure a specific construct. The construct is measured through 5 items on a 5 points likert scale. However, I would probably need to change the ...
0
votes
1answer
84 views
If the Pearson correlation coefficient is small/weak, can it have criterion validity?
The r value I have obtained is -0.190 and p = 0.410.
As scores on stress increase, scores on eating habits decrease.
I know from my textbook that criterion validity is about whether your measure is ...
0
votes
1answer
33 views
Validate 10% sample from population
I have two databases which include amount of passengers between two routes.
One is the full dataset, while the other is supposedly a 10% sample.
So for example, the full database will display a route ...
2
votes
1answer
200 views
How would you validate a new test without a gold standard
I am measuring a variable (continuous, unknown distribution) on a new assessment tool. There is no gold standard to compare to. I can take 2 measures per participant only. There is no intervention ...
3
votes
1answer
107 views
Teaching examples for internal validity: published experiments later shown to be confounded in an interesting way
There are some good examples of interesting/fun spurious correlations here (Examples for teaching: Correlation does not mean causation). Many/most of these are "funny" examples that aren't likely to ...
2
votes
2answers
473 views
Why does a factor need to be internally consistent?
I'm wondering exactly what the question says: Why does a factor (as linear combination of its items) need to be internally consistent?
If I want to use a factor found in a factor analysis as a ...
0
votes
0answers
30 views
Are reliability and validity measures necessary for all parts of a survey
I am conducting analysis on a questionnaire which was a three-part study, not three separate studies. I asked the same exact set of validated scale questions for specific constructs for three ...
0
votes
1answer
31 views
Validity / reliability of instrument used by 2 people at 2 sites
How can the validity/reliability of a semi-structured interview tool be determined, conducted by two persons at two different sites separately? For instance, one person is using the tool with male ...
3
votes
1answer
115 views
Survival analysis: Excluding cases of trivial follow-up time?
I am finishing up the data collection for my retrospective cohort study, which I have designed with the intention of conducting survival analysis. Our outcomes of interest are recurrence, metastasis, ...
