1
$\begingroup$

Surveys like PISA, TIMSS etc. do not report individual achievement scores per observation, but five (sometimes ten) so-called "plausible values", drawn from a constructed probability distribution that represents the students achievement.

The general way to analyse this (as given in many papers in technical reports) is to do five analyses on each set of PVs and then calculate the mean of interesting descriptive statistics (mean, variance etc.)

How do you proceed when doing Bayesian analysis? I suppose you do the analysis five times, get five posterior probability distributions, and then? How do you combine those five distributions?

$\endgroup$
5
  • $\begingroup$ Do you know anything about the distribution from which these plausible values are drawn? $\endgroup$
    – einar
    Commented Nov 13, 2023 at 12:37
  • 1
    $\begingroup$ Honestly, the "methods and procedures" documentation[1] goes over my head. :-) It says "In summary, the plausible values used in TIMSS and other large-scale assessments are random draws from a conditional normal distribution [...] that depend on response data x_n as well as context information z_n estimated using a group-specific model for each country g." $\endgroup$
    – Thomas
    Commented Nov 13, 2023 at 13:05
  • 1
    $\begingroup$ (from [1] Martin, M. O., von Davier, M., & Mullis, I. V. S. (2020). Methods and Procedures: TIMSS 2019 Technical Report. In International Association for the Evaluation of Educational Achievement. International Association for the Evaluation of Educational Achievement. eric.ed.gov/?id=ED610099 $\endgroup$
    – Thomas
    Commented Nov 13, 2023 at 13:06
  • $\begingroup$ And what kind of model would you do five times for five plausible values? Are they predictors in a regression, or the response perhaps? $\endgroup$
    – einar
    Commented Nov 15, 2023 at 13:17
  • 1
    $\begingroup$ They are the response: math (and other) scores in case of PISA and TIMSS. $\endgroup$
    – Thomas
    Commented Nov 16, 2023 at 14:02

1 Answer 1

2
$\begingroup$

I'm no expert in this kind of analysis, but I will sketch out a sort of answer. My feeling is that doing five models isn't the most efficient way of doing this because you're not using the information that the plausible values come from the same distribution (for the same individual).

As I see it you have some pseudo-response values that are drawn from a distribution centered at the true response value. You know that this distribution is a normal distribution. It is fairly straight-forward to use this information in a Bayesian model.

The main idea is to treat the true response variable that you're interested in as an unknown parameter in the model you're estimating, and the plausible values are treated as data to inform on this parameter. I provide an example in R and Stan. You might be able to do this sort of thing with some of the friendlier packages like brms but I wouldn't know how.

First we generate some fake data

set.seed(2023-11-17)

# True model is y = 1 + x + noise
x = rnorm(50)
y = 1 + x + rnorm(50, sd=.05)

We observe some true $y$ values generated as above. For some reason (privacy perhaps?) we choose to provide some randomly drawn plausible values instead of the true value so that the plausible values are centered on the true value:

y_plausible = t(sapply(y, function(yy) { rnorm(5, mean=yy, sd=.025) }))
head(y_plausible)
#>           [,1]      [,2]      [,3]      [,4]      [,5]
#> [1,] 2.1016310 2.0761254 2.1356810 2.1256719 2.0619211
#> [2,] 0.7671114 0.7383365 0.7434291 0.7278975 0.7367159
#> [3,] 1.0430924 1.0673116 1.1229851 1.0795531 1.0617507
#> [4,] 1.4613296 1.4723853 1.4703300 1.4365836 1.4870355
#> [5,] 2.7055201 2.6943164 2.7165702 2.7290942 2.6345115
#> [6,] 0.6336314 0.6393465 0.6268552 0.6292515 0.6270424

This is all that the analyst has to work with, the true $y$ being hidden.

Then formulate a Bayesian model for the data generating process

Below is the code from the file model.stan that I use further down. Basically it says that true $y$, an unknown parameter, is distributed as normal in a typical linear regression way. "Observed" data, $y_{plausible}$, is distributed as normal, centered on true $y$.

data {
  int<lower=0> N;
  vector[N] x;
  vector[5] y_plausible[N];
}

// The parameters accepted by the model. Notice that true obscured y becomes 
// a parameter to be estimated
parameters {
  // regression parameters
  real beta_0;
  real beta_1;
  real<lower=0> sigma;
  
  // parameters of the distributions from which plausible values came. 
  // you can give them individual standard deviations if you want. I just do a
  // common one to all of them, which probably isn't very realistic
  vector[N] y;
  real<lower=0> sigma_plausible;
  
}

// The model to be estimated. 
model {
  y ~ normal(beta_0 + beta_1*x, sigma);
  
  for (i in 1:N) {
    y_plausible[i] ~ normal(y[i], sigma_plausible);
  }
  
  // I just let stan set some default priors, which is probably not advisable in
  // general but for this simple model it seems to work fine
}


Now we fit the model and look at some estimates

Our posterior distribution for the regression slope is more or less centered on the true value of 1:

library(rstan)

dataset = list(N=50, x=x, y_plausible=y_plausible)
stan_fit = stan("model.stan", data=dataset)

draws = extract(stan_fit)

# the intercept, truth marked as a vertical bar
hist(draws$beta_1, breaks = 50, prob=T, border = "lightgrey", 
     main="Draws from beta_1 posterior")
abline(v=1, lwd=1.5)

The posterior for the first true $y$ is also pretty close:

# Let's look at the first y value, truth marked as vertical bar
estimated_y = draws$y
hist(estimated_y[, 1], breaks = 50, prob=T, border = "lightgrey", 
     main="Draws from y[1] posterior")
abline(v=y[1], lwd=1.5)

Created on 2023-11-17 with reprex v2.0.2

$\endgroup$
3
  • $\begingroup$ Thank you very much! I'll have to play a bit with R, but you gave me good material to work with. $\endgroup$
    – Thomas
    Commented Nov 18, 2023 at 15:06
  • 1
    $\begingroup$ Happy to help. If you want the code in easier-to-use format I put it on GitHub: github.com/3inar/plausible_values $\endgroup$
    – einar
    Commented Nov 18, 2023 at 17:13
  • 1
    $\begingroup$ Also (@Thomas in case you don't get notified of replies without it): I was thinking about comparing with the five models approach. What I think you'd do is to fit the five models and then simply put all the posterior draws in one big array. It would be the equivalent of making a mixture of these distributions with equal weights. In the end I leave it as an exercise for the reader. $\endgroup$
    – einar
    Commented Nov 18, 2023 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.