Questions tagged [validity]

An instrument is valid if it measures what it purports to measure. This is different from reliability, which means that an instrument measures (whatever it measures) in a way that is repeatable. DO NOT use this tag for `validation` model purposes.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Is it correct to calculate Average Variance Extracted (AVE) and Composite Reliability (CR) using loadings from exploratory factor analysis?

In order to obtain evidence of convergence and discriminant validity of a scale, I used SPSS to run a Exploratory Factor Analysis; and used its loadings to calculate the Average Variance Extracted (...
user avatar
0 votes
0 answers
7 views

How many times should construct validity undertaken?

When we have done empirical validity such as construct validity of scale items and we have obtain the valid instrument, could we test it again perhaps once or twice on other samples? Perhaps there are ...
user avatar
  • 1
1 vote
1 answer
23 views

Reliable methods to validate clustering of text phrases?

Question is in the title. I have clustered the word embeddings of text phrases, and now want to try and check whether the resulting clusters are coherent enough. I have tried methods that are ...
user avatar
0 votes
0 answers
19 views

How do I do discriminant validity

How do I do discriminant validity the same as in this paper Page #333 "A Technological Pedagogical Content Knowledge (TPACK) Scale for Geography Teachers in Senior High School" https://www....
user avatar
  • 1
0 votes
0 answers
67 views

Discriminant Validity - Chi Square Test

To check discriminant validity between two constructs for CFA in AMOS, I am using chi square difference test to confirm discriminant validity which "can be assessed for two estimated constructs ...
user avatar
  • 1
0 votes
0 answers
11 views

Formally quantifying and assessing difference in calibration scores among different estimators?

Given a number of estimators $f_{1}, \ldots, f_{n}$ and a number of corresponding calibration scores $s_{1}, \ldots, s_{n}$ for each data group $g_{1}, \ldots, g_{m}$ with $m \ll n$. We would like to ...
user avatar
1 vote
0 answers
33 views

Why is internal consistency a measure of reliability?

Why is 'internal consistency' a measure of reliability? It looks like a measure of validity instead. Definitions: "Internal consistency reflects the extent to which items within an instrument ...
user avatar
  • 105
1 vote
0 answers
41 views

How to meta-analyze incremental validity

Suppose we have two instruments (A and B) and we want to meta-analyze whether instrument B has incremental validity over instrument A in predicting an outcome Y. What effect size would be most ...
user avatar
0 votes
0 answers
27 views

psych package: reliability, should complete cases be used?

Would like to seek advice on instrument validation and the psych package in r. I would like to evaluate the reliability first (want to replicate the study here: https://doi.org/10.1186/s12885-018-5135-...
user avatar
  • 165
0 votes
0 answers
21 views

Instrument validation: Proportional-odds ordinal regression?

I am looking for advice on the best statistical tests to use to accomplish my goal. I am making a new instrument to measure a variable and want to compare it to the current (gold standard) instrument. ...
user avatar
0 votes
0 answers
9 views

Term for how well a measure correlates with measures of related but different constructs

Is there a term that refers to how well a measure correlates with measures of related but different constructs? I suspect that there should be some such term, probably including the word "...
user avatar
  • 695
0 votes
0 answers
46 views

colinearity in stepwise regression

I would be very grateful if you can be so kind to clarify what may be wrong in the following steps and interpretation of the validity conditions of the stepwise regression. I am afraid there is a ...
user avatar
  • 1
1 vote
2 answers
50 views

I want to test for a significant difference in the result (t-value or effect size) of two paired t-tests

I am testing the quality of a measurement construct, where I propose that Measure A is better (i.e. more responsive to situational changes) than Measure B is. I conduct a study in which I measure the ...
user avatar
  • 11
0 votes
0 answers
6 views

What is the best way to handle longitudinal data where the follow-up question for the same outcome/construct has changed

I'm doing secondary analysis on a large longitudinal dataset (didn't collect myself) and have a series of questions related to movement/time-use behaviours (sleep, physical activity, sedentary ...
user avatar
0 votes
0 answers
15 views

Has a valid kernel to be positive in one dimension?

During my ML exercises, I have come across the kernel below: $$ k(x,y) = sin(x)sin(y),\ x,y \in \mathbb{R}$$ The text is asking me if this is a valid kernel. My current knowledge would reply "yes&...
user avatar
  • 101
0 votes
1 answer
20 views

Validity of multilevel modeling to include results for multiple psychometric tests with subscales #statsnube

I have survey data for approx n = 1650 for multiple psychometric tests (all participants have completed all the tests), and about 12 outcome variables (from a PCA of a 52-question survey, e.g. ...
user avatar
1 vote
1 answer
33 views

Instrument validity: does a positive and significant coefficient on Z in a regression of Y on X and Z pose a problem?

I have an initial regression of Y on X and Z. Both of my coefficients on X and Z are non-zero and strongly statistically significant. X and Z are correlated but I am told collinearity shouldn't be an ...
user avatar
  • 11
1 vote
1 answer
45 views

Confirmatory Factor Analysis : validity for a just-identified model

I have developed a 3-item questionnaire, and I would like to test the validity of this questionnaire. I wanted to test the validity using confirmatory factor analysis, but since my model is just-...
user avatar
0 votes
0 answers
51 views

Testing validity of an instrumental variable with exact identification

I am conducting an instrumental variable regression. I have one endogenous variable and one instrument? How can I test if the instrument is valid? What I have done for now is run the ivreg command and ...
user avatar
1 vote
1 answer
59 views

Discussing validity of tests performed after a cluster analysis

I'm new to datascience (from a medical/medical science background). My supervisor (social sciences background) asked me to assist in rewriting a paper where we do a cluster analysis for a ...
user avatar
2 votes
0 answers
163 views

2SLS approach using lagged dependent variable as an instrument

I was wondering if the lagged dependent variable can be a valid instrument or not. In my data, policy intervention (x) is made depending on the outcome (y) in the past. (not depending on the effect (b)...
user avatar
1 vote
0 answers
25 views

Propensity score matching at the group level and models at the subject level

I am running a study that uses a combination of propensity score matching and mixed-effects models. I use propensity score matching to locate some groups (half of them represent control, and another ...
user avatar
2 votes
0 answers
24 views

2stage (instrumental variable) analysis: is reverse causality in the first stage a problem? [duplicate]

For my research I am looking into the relationship between an outcome (Y) and a predictor (X) as follows: $$Y = X + e$$ where $e$ is the error term. Because there might be reverse causality I am ...
user avatar
  • 73
0 votes
2 answers
74 views

Is this an instance of convergent evidence?

In psychometrics, convergent validity is a source of proof for construct validity. It is said that it is the degree to which test scores are correlated with test scores of relevant constructs or the ...
user avatar
  • 1
0 votes
0 answers
54 views

Validity with repeated measures

I have some data that was collected with 2 different devices on a sample of 50 subjects. It was also collected for 50 days in a row. One of the devices is considered "the gold standard" ...
user avatar
  • 1
0 votes
0 answers
27 views

Algorithm to calculate validity of document?

I have a dataset of text-documents, which are all considered valid. Also I have a set of manually defined features that are required to be in the document for it to be considered valid. The features ...
user avatar
1 vote
1 answer
769 views

Placebo test for the validity of difference-in-differences analysis results for limited data period

I am doing an analysis of a strategy implemented by a four-year-old corporate company and this implementation was given in different cities which all of its employees were received the treatment or ...
user avatar
0 votes
1 answer
19 views

Can reliability and validity assessments be considered as required constituent of the "Measurement part" in Structural Equation Modeling?

I wonder if any type of reliability and / or validity analysis can be considered as an element of the Measurement part in SEM. For example, when computing Cronbach's alpha or the AVEs of an instrument,...
user avatar
1 vote
0 answers
9 views

Measuring Validity and reliability

Say you have an independent train ticket system where automatic refunds would be issued in case of a delayed arrival. To determine the arrival time of the train at a certain station you collect data ...
user avatar
  • 111
1 vote
1 answer
46 views

Correlation between questionnaire to test validity

I'm doing a validation of a questionnaire for my thesis and I should investigate the validity of my test with others one. To do so I've been asked to calculate the correlation between two scales and I'...
user avatar
0 votes
0 answers
24 views

Questionnaire validity statistical significance

My colleagues and I, have developed a questionnaire of 116 questions, segmented in 7 constructs. Cronbach's a indices for each construct are above 70%, and it is content valid. However, when we apply ...
user avatar
2 votes
0 answers
427 views

Comparison between two measurement devices

I have two measurement devices accelerometer A and B. The measurement is the step time of a person walking in a corridor. A is the gold standard. I want to check the validity and reliability of B. how ...
user avatar
0 votes
0 answers
23 views

Survey Question Validity

I have a survey that asks the user to report their debt from their undergraduate education and their debt from their graduate education. I intend to add these two averages to measure their average ...
user avatar
0 votes
0 answers
26 views

Generalizability study of subscaled rating instrument

I have a school rating ($1$ thru $4$) instrument that consists of $9$ subscales (e.g., classroom instruction, school management etc.). Under each subscale, I have $6$ items on which rating occurs. I ...
user avatar
  • 3,076
0 votes
1 answer
27 views

Is it always needed to do a validity test for variables?

Let's say I have one variable called "perceived image", which is measured with 6 items (questions) that I adapted from previous literature. If I have a Cronbach's alpha > 0.7 (for reliability), do I ...
user avatar
  • 7
3 votes
1 answer
152 views

Do I estimate factor loadings in a confirmatory factor anlysis (CFA) aimed at verifying an exploratory factor analysis (EFA)?

I decided to use a questionnaire published by another researcher (paper and supplementary here). In the article they perform an EFA, find two factors, and report the resulting factor loadings (...
user avatar
0 votes
0 answers
49 views

How to test nomological validity with lavaan?

in my understanding, I need to test nomological validity with SEM. But what exactly should I look at in e.g. summary(fit1, standardized=TRUE, fit.measures=TRUE)? Code for reference: ...
user avatar
  • 539
0 votes
1 answer
116 views

Validity of ARDL regression when there is I(2) variable

I am trying to model relationship between Y(t) and Y(t-1), X(t), X(t-1) and Z(t) using ARDL model. Most cases these time-series are I(0) or I(1). However sometimes I encounter I(2) or higher order. ...
user avatar
1 vote
1 answer
2k views

item-total correlation vs. inter-item correlation

I am assesing the reliability and validity of an 16-item multuiple choice test supposed to test language skills. Cronbach's apha is "questionable" with .65. I am having troubles interpreting the ...
user avatar
  • 779
0 votes
0 answers
258 views

Wilcoxon signed-rank test for paired and repeated measures data

I have a dataset with 45 participants, each with 1-7 datapoints/observations (126 total observations). Each of these observations represents a window of time in which two different methods attempt to ...
user avatar
  • 1
4 votes
0 answers
65 views

How can statistics be used to avoid "Lending False Credibility To Decisions We've Already Made"

In light of this article Data Science Has Become About Lending False Credibility To Decisions We've Already Made published in Forbes, I would appreciate input from the statistical and data science ...
1 vote
1 answer
19 views

Are there any studies of generalized error performance that don't assume data quality is constant with sample size?

As far as I know, much of the statistical and machine learning literature where modeling algorithms are compared for their generalization error performance as a function of sample size (think of ...
user avatar
0 votes
0 answers
14 views

factorial designed study within between

IV1 : alcohol use, IV2: breathing exercise (both of them within subject variable) IV3: gender (between subject variable). My research question is that: ''Does alcohol usage and breathing exercise's ...
user avatar
0 votes
1 answer
182 views

Calculating Confidence Interval with log-transformation

I fitted a linear regression with a 3rd order interaction and adjustment for confounding variable which looks like this: $$ log(Y) = β_1S + β_2T + β_3M + β_4ST + β_5SM + β_6TM + β_7STM + β_XX $$ ...
user avatar
  • 1,119
0 votes
1 answer
18 views

Can findings based on low values be considered valid?

Lets say i do have a TimeSeries with N > 300 but values like [0, 1, 2, 1, 2, 0, 2, ...] representing the visitors count of a website per day. Since there are only few visitors and each of them can be ...
user avatar
  • 1
3 votes
1 answer
123 views

What does "the hypotheses ... are verified as null" mean?

Context: This article examines the criterion-related validity of the results obtained by the application of four different methods of assessment to the correction of a second-year exam of ...
user avatar
  • 653
2 votes
2 answers
377 views

Reweighting features in PAM/K-means clustering

As stated in Hennig et al. 2016 Handbook of cluster analysis: If for subject matter reasons some variables are more important than others regardless of the within-variable variation, one could ...
user avatar
  • 185
9 votes
3 answers
937 views

Does a statistic used as a measure becomes invalid after it's reported on? [closed]

Apologies if this the wrong place to ask. but i'm trying to explain to a director that his Statistics don't show the truth because he is demanding that the stats are what is reported on and staff are ...
user avatar
0 votes
1 answer
110 views

How to assess predictive validity of a variable on the outcome?

I am looking to assess the predictive validity of two measures (M1 and M2) on a binary outcome. After constructing my model, do I just calculate odds ratios and compare the odds ratios for the two ...
user avatar
1 vote
0 answers
60 views

Measuring emotion differentiation on a within-person level.

To measure emotion differentiation, which refers to the precision of an individual to accurately describe and distinguish between different emotional states, many studies have used a between-person ...
user avatar
  • 61