Questions tagged [validity]

An instrument is valid if it measures what it purports to measure. This is different from reliability, which means that an instrument measures (whatever it measures) in a way that is repeatable. DO NOT use this tag for validation model purposes.

136 questions
Filter by
Sorted by
Tagged with
1 vote
10 views

Clustering and samples weighting (using GEE)

I am currently analyzing the social factors associated with the prevalence of infectious diseases in European countries using GEE models. My data is based on patients covered by a specific health ...
12 views

calculation of the C-index clustering for manual [duplicate]

Can anyone give me an example of working on the C-index clustering validity test, but calculating manually??
• 1
42 views

Validating binary prediction model

Suppose we have a model that predicts for binary event $e$ ($0$ or $1$) with a single output $p$ (the expected probability $e$ occurs). If we are able to compare $p$ with the true value of $e$ ($0$ or ...
• 161
8 views

Can you build up statistical validity with multiple month's worth of the same survey questions?

The company I work for conduct consulting where we analyse company survey responses for statistical validity against the general company population. To prevent survey fatigue, the company sends out ...
57 views

In a regression, is an independent variable still useful if it is the same for each subject between observations?

I was not able to find an answer to this question, as I feel it is rather specific and awkward to ask. But, here goes. This is a simplified version of my data: As you can see, I have various subjects ...
• 21
11 views

Validity, reliability and repeatability

I want to validate an anemometer in a wind tunnel VS gold standard. Thinking in validity, reliability and repeatability. The dependent variable is wind speed. I can test 3 different units of the same ...
41 views

Validation Checklist Passing Score

I have a mixed-method thesis ongoing. I'm currently on the validation stage of my survey and interview questions. I have adopted a validation checklist from the internet composing of 15 items with a 4-...
• 41
165 views

What is the (logical) relationship between an measurement instrument's validity and responsiveness?

The three major criteria for the quality of a measurement instrument are validity, reliability, and responsiveness. I understand validity and reliability, but I have trouble understanding the need to ...
• 131
1 vote
1k views

What is meant by practical significance and statistical significance? How do we distinguish these concepts? [duplicate]

I am unable to understand content reported in : Practical vs Statistical significance Statistical significance-testing implies whether a sample-statistic matches with population estimate of effect-...
10 views

Validating a new definition for detection

I am having some trouble understanding what approach I would use to validate a new set of definitions versus the "gold standard" to detect disease positivity. For example, if I wanted to say ...
• 75
62 views

Difference between kappa statistics

I am currently developing an instrument and have some questions on the use of kappa statistics for interrater agreement used for content validity. I have gone through previous literature and I still ...
• 31
22 views

Estimate the threohold of reject region by Bootstrap

I am studying the Bootstrap method and encountered a question about how to estimate the reject region. Suppose we have observed a sample $\{(X_i, Y_i)\}_{i=1}^n$, I aim to investigate the linear ...
• 43
1 vote
20 views

Approach of a Golden Standard Measurement with questionnaire data

I am a bit lost on the correct approach to this Statistical problem. Let me explain the context of this problem: I have a cohort population with longitudinal measurements during a period of several ...
• 11
140 views

Is it appropriate to use log-rank test in a study having multiple cases with expected values less than 5?

My goal is to show independency of factors related to survival of a small size sample (26 patients) for a period of time, this sample was divided in 2 groups (16 vs 10 patients), the events were ...
• 37
299 views

Can I check the reliability of my test after collecting my data?

I have performed a study where I tested 2 groups, before and after two different types of manipulations. The main outcome of my study were the group differences at time point 2. Now someone asks what ...
• 309
1 vote
10 views

Gender Differences in Validity

I need to demonstrate discriminant validity for two subscales (subscales have 6 and 7 items) that should differ between males and females. There is a theoretical basis that there should be a ...
• 11
38 views

The validity of using paired sample t-test to compare results from two different test methods

My research involved two types of contextualized tests to examine two types of knowledge of collocations. The first test is a multiple-choice (with 3 options) and it was used to examine receptive ...
• 41
1 vote
31 views

Gender Differences in Subscales [closed]

I need to demonstrate discriminant validity for two subscales (subscales have 6 and 7 items) that should differ between males and females. There is a theoretical basis that there should be a ...
• 11
1 vote
26 views

What is the difference between validity testing and actual results?

Researches usually present the results of validity tests conducted before the actual results - these include descriptive statistics with Mean and SD and also inter-construct correlations. Then there ...
• 31
1 vote
84 views

Reliable methods to validate clustering of text phrases?

Question is in the title. I have clustered the word embeddings of text phrases, and now want to try and check whether the resulting clusters are coherent enough. I have tried methods that are ...
• 11
1 vote
92 views

Why is internal consistency a measure of reliability?

Why is 'internal consistency' a measure of reliability? It looks like a measure of validity instead. Definitions: "Internal consistency reflects the extent to which items within an instrument ...
• 485
1 vote
69 views

How to meta-analyze incremental validity

Suppose we have two instruments (A and B) and we want to meta-analyze whether instrument B has incremental validity over instrument A in predicting an outcome Y. What effect size would be most ...
• 65
41 views

Instrument validation: Proportional-odds ordinal regression?

I am looking for advice on the best statistical tests to use to accomplish my goal. I am making a new instrument to measure a variable and want to compare it to the current (gold standard) instrument. ...
98 views

colinearity in stepwise regression

I would be very grateful if you can be so kind to clarify what may be wrong in the following steps and interpretation of the validity conditions of the stepwise regression. I am afraid there is a ...
1 vote
71 views

I want to test for a significant difference in the result (t-value or effect size) of two paired t-tests

I am testing the quality of a measurement construct, where I propose that Measure A is better (i.e. more responsive to situational changes) than Measure B is. I conduct a study in which I measure the ...
• 11
59 views

Validity of multilevel modeling to include results for multiple psychometric tests with subscales #statsnube

I have survey data for approx n = 1650 for multiple psychometric tests (all participants have completed all the tests), and about 12 outcome variables (from a PCA of a 52-question survey, e.g. ...
• 11
1 vote
104 views

Instrument validity: does a positive and significant coefficient on Z in a regression of Y on X and Z pose a problem?

I have an initial regression of Y on X and Z. Both of my coefficients on X and Z are non-zero and strongly statistically significant. X and Z are correlated but I am told collinearity shouldn't be an ...
• 11
1 vote
299 views

Confirmatory Factor Analysis : validity for a just-identified model

I have developed a 3-item questionnaire, and I would like to test the validity of this questionnaire. I wanted to test the validity using confirmatory factor analysis, but since my model is just-...
515 views

Testing validity of an instrumental variable with exact identification

I am conducting an instrumental variable regression. I have one endogenous variable and one instrument? How can I test if the instrument is valid? What I have done for now is run the ivreg command and ...
1 vote
137 views

Discussing validity of tests performed after a cluster analysis

I'm new to datascience (from a medical/medical science background). My supervisor (social sciences background) asked me to assist in rewriting a paper where we do a cluster analysis for a ...
• 11
386 views

2SLS approach using lagged dependent variable as an instrument

I was wondering if the lagged dependent variable can be a valid instrument or not. In my data, policy intervention (x) is made depending on the outcome (y) in the past. (not depending on the effect (b)...
1 vote
129 views

Propensity score matching at the group level and models at the subject level

I am running a study that uses a combination of propensity score matching and mixed-effects models. I use propensity score matching to locate some groups (half of them represent control, and another ...
28 views

2stage (instrumental variable) analysis: is reverse causality in the first stage a problem? [duplicate]

For my research I am looking into the relationship between an outcome (Y) and a predictor (X) as follows: $$Y = X + e$$ where $e$ is the error term. Because there might be reverse causality I am ...
• 73
154 views

Is this an instance of convergent evidence?

In psychometrics, convergent validity is a source of proof for construct validity. It is said that it is the degree to which test scores are correlated with test scores of relevant constructs or the ...
• 1
108 views

Validity with repeated measures

I have some data that was collected with 2 different devices on a sample of 50 subjects. It was also collected for 50 days in a row. One of the devices is considered "the gold standard" ...
29 views

Algorithm to calculate validity of document?

I have a dataset of text-documents, which are all considered valid. Also I have a set of manually defined features that are required to be in the document for it to be considered valid. The features ...
• 251
1 vote
2k views

Placebo test for the validity of difference-in-differences analysis results for limited data period

I am doing an analysis of a strategy implemented by a four-year-old corporate company and this implementation was given in different cities which all of its employees were received the treatment or ...
• 11
1 vote
70 views

Can reliability and validity assessments be considered as required constituent of the "Measurement part" in Structural Equation Modeling?

I wonder if any type of reliability and / or validity analysis can be considered as an element of the Measurement part in SEM. For example, when computing Cronbach's alpha or the AVEs of an instrument,...
• 171
1 vote
11 views

Measuring Validity and reliability

Say you have an independent train ticket system where automatic refunds would be issued in case of a delayed arrival. To determine the arrival time of the train at a certain station you collect data ...
• 111
1 vote
233 views

Correlation between questionnaire to test validity

I'm doing a validation of a questionnaire for my thesis and I should investigate the validity of my test with others one. To do so I've been asked to calculate the correlation between two scales and I'...
26 views

Questionnaire validity statistical significance

My colleagues and I, have developed a questionnaire of 116 questions, segmented in 7 constructs. Cronbach's a indices for each construct are above 70%, and it is content valid. However, when we apply ...
• 73
1k views

Comparison between two measurement devices

I have two measurement devices accelerometer A and B. The measurement is the step time of a person walking in a corridor. A is the gold standard. I want to check the validity and reliability of B. how ...
28 views

Survey Question Validity

I have a survey that asks the user to report their debt from their undergraduate education and their debt from their graduate education. I intend to add these two averages to measure their average ...
• 159
28 views

Generalizability study of subscaled rating instrument

I have a school rating ($1$ thru $4$) instrument that consists of $9$ subscales (e.g., classroom instruction, school management etc.). Under each subscale, I have $6$ items on which rating occurs. I ...
• 4,046
68 views

Is it always needed to do a validity test for variables?

Let's say I have one variable called "perceived image", which is measured with 6 items (questions) that I adapted from previous literature. If I have a Cronbach's alpha > 0.7 (for reliability), do I ...
• 7
278 views

Do I estimate factor loadings in a confirmatory factor anlysis (CFA) aimed at verifying an exploratory factor analysis (EFA)?

I decided to use a questionnaire published by another researcher (paper and supplementary here). In the article they perform an EFA, find two factors, and report the resulting factor loadings (...
92 views

How to test nomological validity with lavaan?

in my understanding, I need to test nomological validity with SEM. But what exactly should I look at in e.g. summary(fit1, standardized=TRUE, fit.measures=TRUE)? Code for reference: ...
• 569
593 views

Validity of ARDL regression when there is I(2) variable

I am trying to model relationship between Y(t) and Y(t-1), X(t), X(t-1) and Z(t) using ARDL model. Most cases these time-series are I(0) or I(1). However sometimes I encounter I(2) or higher order. ...
1 vote
5k views

item-total correlation vs. inter-item correlation

I am assesing the reliability and validity of an 16-item multuiple choice test supposed to test language skills. Cronbach's apha is "questionable" with .65. I am having troubles interpreting the ...
• 909