2
$\begingroup$

Originally, a test battery had 4 components: two 100 item multiple-choice tests, one oral test, and one essay test. Each component measured different topics. Each of the 4 components was weighted 25%.

Now the test length of each multiple-choice test has been reduced to 60 items (for practical reasons). Each of the components is still weighted 25%.

If the reliability of the two MC tests had been reduced to zero, the weight given to the two topics measured by the two MC tests would be zero. (The two MC tests would be contributing only error.) In actuality, the reliability was reduced, but not to zero.

My question is, what did that change in reliability do to the weight given to the two topics measured by the two MC tests. How can I estimate the difference in weight due to the change?

Clarification: The original weights were chosen based on a consensus of interested parties. Therefore, I start with the premise that the weighting of the original test battery is ideal (insofar as the parties agreed on those weights). The reduction in number of items was due to expediency. Fewer items results in lower test reliability. I seek to determine how the new test battery with the shorter MC tests compares to the original in terms of the weight given to the topics measured by each of the tests in the battery. (It seems intuitively obvious that the less reliable tests result in less effective weight given to the topics measured by those tests. In the extreme, if the tests had zero reliability, they would add only random noise to the overall score of the test battery, the nominal weight of 25% notwithstanding, and the topics tested by the MC tests would not have any weight at all.)

Update to address comments:

It seems that my question contains some unsophisticated or non-standard vocabulary. So, here is another attempt to state my question, using other words.

The original weights of 25% assume an existing level of validity of each of the four tests and the concensus is that these tests should, ideally, be weighted equally. I take this to mean that each test originally contributed 25% of the non-random, valid variance to the total score.

I presume that the switch to shorter MC tests has changed the reliablity of the two MC tests and, therefore, also their validity. So the shorter MC tests are contributing less valid variance to the total score. In that sense, they are no longer weighted 25%. My question is, what is their new weight.

Two Possible Approaches: Here is one thought on how I might approach the question I posed. Validity is capped by the square root of reliability. So, if the reliability is reduced from .8 to .7, the validity cap goes from .89 to .84. That seems to imply that the valid contribution of the new, shorter MC test is 5% less than the original test. In terms of proportions, that is .05/.89 less, or a new lower weight about 6% less than the original weight. That is, the shorter MC test is contributing 6% less valid information to the total score. Perhaps one could compensate for that by adjusting the 25% weights. Any thoughts on this approach?

Alternatively, since reliability is linearly related to utility, perhaps a ratio of the reliabilities is a better index of the reduced impact of the MC tests. That might be .7/.8=.875, or a reduction in weight of 12.5%. Any thoughts on this approach?

$\endgroup$
15
  • $\begingroup$ As apparent already in the discussion regarding the answer by @jginestet, something is unclear here. You don't say how exactly (according to which rationale) the weights are chosen. Your later question implies that the weight choice has something to do with reliability, but your earlier statement (25% in both situations) seems to suggest that this isn't the case. $\endgroup$ Commented Jul 9 at 9:21
  • 1
    $\begingroup$ Maybe you believe that the weight choice has to be done in an automatic ("objectively correct"/unified) way depending on reliability, but this is not the case. There may be different considerations that can influence the weight choice other than reliability. This means that from your question it isn't clear to us how the weights are actually chosen in this case. And thus the information given is not enough to answer the question. $\endgroup$ Commented Jul 9 at 9:23
  • $\begingroup$ TY. The original weights were chosen based on a consensus of interested parties. The change in weights was due to expediency. I seek to determine how the new test battery with the shorter MC tests compares to the original in terms of the weight given to each of the tests in the battery. $\endgroup$
    – Joel W.
    Commented Jul 9 at 13:24
  • $\begingroup$ "The change in weights was due to expediency." What change in weights? Weights are only given in the first two paragraphs, and they are apparently not changed. $\endgroup$ Commented Jul 9 at 14:41
  • 1
    $\begingroup$ @DavidB I added an update to the question to try to address your comment. $\endgroup$
    – Joel W.
    Commented Jul 11 at 18:22

2 Answers 2

1
+50
$\begingroup$

I do not think it's possible to approach this question the way you suggest. However, one way to get at what I think might be your underlying question is with structural equation modeling (SEM), and more specifically with a confirmatory factor analysis (CFA). The way the total score is constructed implies that there is some latent property that each test partially measures, and by averaging them we seek to 'pull-out' that latent property. With a CFA we can explicitly model this relationship, including testing whether all 4 tests equally contribute to the latent property, whether there is some imbalance, and whether the imbalance changes when using different indicators (i.e., lower test reliability). (NB, as a previous poster mentioned, simply because a test is shorter is no guarantee that it is less reliable. Indeed, by making the test shorter you could have removed less reliable items, which would make the overall test more reliable).

A basic overview of what I mean:

In R:

Load in some packages and add a function for simulating reliability

library(lavaan)
library(MASS)
library(dplyr)
library(lavaanPlot)
library(semTools)

## function to add noise to a variable
## rel = reliability

add.noise = function(x,rel){
  x2 = base::scale(x = (x+stats::rnorm(length(x), 0, sqrt((1-rel)/rel))),
                   center = TRUE, scale = TRUE)
  attributes(x2) = NULL
  return(x2)
}

Simulate four correlated variables

set.seed(34736)

# make a correlation matrix

cor.mat = matrix(0,nrow = 4,ncol = 4)
cor.mat
cors = rnorm(n = 6,mean = .5,sd = .125)
cor.mat[lower.tri(cor.mat)] = cors
cor.mat = as.matrix(as.dist(cor.mat))
diag(cor.mat) = rep(1,4)

> cor.mat
          1         2         3         4
1 1.0000000 0.4048900 0.4350827 0.5618875
2 0.4048900 1.0000000 0.4460842 0.7136290
3 0.4350827 0.4460842 1.0000000 0.5719406
4 0.5618875 0.7136290 0.5719406 1.0000000

Sample 1000 subjects from these variables and add some measurement error

dat = MASS::mvrnorm(n = 1000,mu = rep(0,4),Sigma = cor.mat ) %>% as.data.frame()
colnames(dat )= c("V1","V2","V3","V4")

cor(dat)

          V1        V2        V3        V4
V1 1.0000000 0.3898022 0.4015777 0.5537640
V2 0.3898022 1.0000000 0.3714067 0.6890184
V3 0.4015777 0.3714067 1.0000000 0.5136934
V4 0.5537640 0.6890184 0.5136934 1.0000000

# add some noise 

dat2 = apply(dat,2,function(X){add.noise(X,.9)}) %>% as.data.frame()

cor(dat2)

          V1        V2        V3        V4
V1 1.0000000 0.3821752 0.3764688 0.5072741
V2 0.3821752 1.0000000 0.3466871 0.6281293
V3 0.3764688 0.3466871 1.0000000 0.4593322
V4 0.5072741 0.6281293 0.4593322 1.0000000

Fit a model with a single factor

model <- ' latent  =~ V4 + V1 + V3 + V2 '
fit <- cfa(model, data = dat2)
lavaanPlot(model = fit,coefs=TRUE)

Single factor model

We see that our four tests (indicator variables) contribute differently to the theoretical shared latent construct.

Add additional measurement to two of the variables and re-fit the model.

 dat3 = dat2
    
    dat3$V1 = add.noise(x = dat[,1],rel = .75)
    dat3$V3 = add.noise(x = dat[,3],rel = .75)

    
    cor(dat3)
    
              V1        V2        V3        V4
V1 1.0000000 0.3443602 0.3080258 0.4659182
V2 0.3443602 1.0000000 0.3111665 0.6281293
V3 0.3080258 0.3111665 1.0000000 0.4049310
V4 0.4659182 0.6281293 0.4049310 1.0000000
    
    model2 <- ' latent  =~ V4 + V1 + V3 + V2 '
    fit2 <- cfa(model2, data = dat3)
        
    lavaanPlot(model = fit2,coefs=TRUE)

Single factor model with more measurement error

We can see that this has slightly decreased the loadings for a couple of our tests.

We can also test whether a sum score where all four tests are equally weighted is an adequate estimate of our theoretical latent property:

model3 <- ' latent  =~ V4 + 1*V1 + 1*V3 + 1*V2 '
fit3 <- cfa(model3, data = dat3)


comp = semTools::compareFit(fit3,fit2)
summary(comp)

Chi-Squared Difference Test

     Df   AIC   BIC    Chisq Chisq diff   RMSEA Df diff Pr(>Chisq)    
fit2  2 10415 10454   9.6802                                          
fit3  5 10527 10552 127.8523     118.17 0.19593       3  < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

####################### Model Fit Indices ###########################
        chisq df pvalue rmsea   cfi   tli  srmr        aic        bic
fit2   9.680†  2   .008 .062† .992† .976† .022† 10415.057† 10454.319†
fit3 127.852   5   .000 .157  .871  .845  .120  10527.229  10551.768 

################## Differences in Fit Indices #######################
            df rmsea    cfi    tli  srmr     aic    bic
fit3 - fit2  3 0.095 -0.121 -0.131 0.097 112.172 97.449

lavaanPlot(model = fit3,coefs=TRUE)

Factor model with equal loadings

In this case, the sum score is a significantly worse fit than letting each variable have it's own weight. However, the difference is low enough that you could probably make a convincing argument that the sum score is an adequate approximation.

$\endgroup$
7
  • $\begingroup$ Pls explain your first sentence. $\endgroup$
    – Joel W.
    Commented Jul 15 at 15:36
  • $\begingroup$ You mean "I do not think it's possible to approach this question the way you suggest"? As it states, I do not know of a way to decompose how much of the true-score variance in a composite score is attributable each of the input variables being combined. $\endgroup$
    – David B
    Commented Jul 15 at 15:53
  • $\begingroup$ If you "do not know of a way to decompose how much of the true-score variance in a composite score is attributable each of the input variables being combined" what would CFA provide? (As to your NB: Since the change to the shorter tests is due to practical considerations, particularly workload of the test writers, I expect the quality of the test items on the newer tests will be no better than the items on the longer tests.) $\endgroup$
    – Joel W.
    Commented Jul 15 at 19:18
  • $\begingroup$ As I said in my answer, I think it gets at something related to what you're interested in, which is whether the amount of shared variance across the measures changes. $\endgroup$
    – David B
    Commented Jul 15 at 20:01
  • $\begingroup$ Why do you say, "I do not think it's possible to approach this question the way you suggest"? Is there nothing logical about the approaches I propose? $\endgroup$
    – Joel W.
    Commented Jul 19 at 14:03
2
$\begingroup$

It depends on how the final score is being computed. But, a priori, no, changing the number of questions on the 2 tests does not change the weights. Someone a priori decided that these 4 components were equally important, and they remain so, no matter how many questions are asked on the multiple choice tests. Just as they would remain the same if the essay went up from 2 pages to 4, or down from 3 pages to 2. Now, if the 2 multiple-choice were scored on a total of 100, and are now scored on a total of 60, while the other 2 tests are still scored on 100, that would change the final score (in fact it would go from a max of 400 to a max of 320), and yes, the other 2 tests would be weighted more. But that would be an odd way to go. Instead, if each test is worth 25, for a total of 100, then whether 100/100 gives you a 25, or 60/60 gives you a 25, the weight would remain the same, with no need for any change. You could argue that it is now a bit easier to get a "perfect" score on the 2 multiple choice tests, but that does not change the weights; it just brings into question whether you are now truly assessing this component properly with only 60 questions, which is a completely different question.
BTW, not sure this has much to do with statistics, but...

$\endgroup$
6
  • $\begingroup$ If the reliability of the two MC tests was reduced to zero, the weight given to the two topics measured by the two MC tests would be zero. In the question posed, the reliability was reduced, but not to zero. My question was, what did that change in reliability do to the weight given to the two topics measured by the two MC tests. $\endgroup$
    – Joel W.
    Commented Jul 7 at 2:12
  • $\begingroup$ Determining the "reliability" of how the 4 components are assessed is a complitely different question than that of the weights. It may be that the reliability of the MC tests has been reduced, or it may have improved! Getting a perfect score on the MC tests with 100 questions may have been much harder than getting 100% on the oral or essay tests? There must have been a reason that the MC questions were reduced, and a validation of sort that it still assessed these 2 components "properly". However, these 2 components still should be weighted equally to the other 2. $\endgroup$
    – jginestet
    Commented Jul 7 at 17:29
  • $\begingroup$ A criterion-related validation study would be very desirable but is impossible. I am seeking a estimate of the impact of reducing reliability on the weight given to the topics tested by the two MC tests. As I said in the (revised) question, if the MC tests were completely unreliable, they would be adding only noise to the test battery score. In that case, the topics meant to be tested by the two MC tests would be contributing zero weight to the test battery score. $\endgroup$
    – Joel W.
    Commented Jul 7 at 17:48
  • 1
    $\begingroup$ Understood. But the fact that the number of questions has been reduced is not demonstrably a reduction in "reliability" (however that is defined). As I previously said, baring further details, it is just as likely that reducing the number of questions has increased the reliability (compared to the other non-MC tests) or that it had no effect whatsoever. What is important is that the 4 tests are "equivalent" in the degree to which they test their component. If one is much harder than the 3 others, then it misrepresents the candidate's ability. $\endgroup$
    – jginestet
    Commented Jul 7 at 23:14
  • $\begingroup$ It is well known that increasing the number of items in a MC test increases the reliability of the test, other things being equal. So your claim that "it is just as likely that reducing the number of questions has increased the reliability" is just wrong. $\endgroup$
    – Joel W.
    Commented Jul 8 at 22:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.