3
$\begingroup$

I am working with the TIMMS and PIRLS data. Both data sets use a Jackknife repeated replication design. Both data sets also use 5 plausible values for each student's grade (5 values with different imputations/models)

I want to compute the Cohen's D and its confidence interval of two groups of students (say girls vs boys).

1) from what I understand from the JRR, the Jackknife is only used to compute the standard error of the mean of a group -- and as far as I understand I do not need the s.e. I need the mean (which is the mean of the weighted mean of each plausible value), and the variance (which is the mean of the weighted variance of each plausible value). Is this right so far?

2) To calculate D, I need to pool the two groups' variances. Here lies my first problem: what are the N for each group (Nboys and Ngirls)? Since the data is weighted, how do I calculate the N for each group?

3) To compute the confidence interval of the D, I am using the MBEES R package which uses the non-central t-distribution technique. Again I need the two N for that. But more to the point, I feel I am cheating somewhere. I just avoided using the whole JRR machinery because I do not need the confidence interval for the mean - but now I am talking of confidence intervals (for the D) shouldn't I use the JRR somewhere??

PS. I found easier to learn about JRR (and BRR) from the R code of the package intsvy which performs the JRR calculations for mean, and for regression confidence (files intsvy.mean.pv.R and intsvy.reg.pv.R)

$\endgroup$

1 Answer 1

1
$\begingroup$

you aren't only asking how to calculate cohen's d with jackknifed survey data, you are asking how to calculate cohen's d with multiply-imputed jackknifed survey data. i doubt that anyone has implemented this.. so i can't really help with that technique if you are dead-set on using it.

if your goal is to simply compare two groups with timss and pirls then i recommend using svycontrast which has been implemented to work with multiply-imputed, jackknifed survey data. to do this, you can load both timss[1] and pirls[1] directly onto your laptop by following those step-by-step instructions. once you've created the complex-sample survey design object that you want, you should be able to modify this snippet of svycontrast code that precisely matches the OECD's PISA methodology (another replication-based and multiply-imputed publicly-available survey dataset).

[1] https://github.com/ajdamico/asdfree/tree/master/Trends%20in%20International%20Mathematics%20and%20Science%20Study

[2] https://github.com/ajdamico/asdfree/tree/master/Progress%20in%20International%20Reading%20Literacy%20Study

[3] https://github.com/ajdamico/asdfree/blob/188018bf3b73ec9ef9c81dd9c1655d78b6007237/Program%20for%20International%20Student%20Assessment/extract%20specific%20countries.R#L131

$\endgroup$
2
  • $\begingroup$ Thanks for the links. 1) do you know any papers that discuss the Cohen's D CI problem for multi-imputed/JRR data? I may be able to implement the code if I knew the theory and /or formulas. $\endgroup$ Commented Apr 6, 2016 at 1:46
  • $\begingroup$ you are in a lonely corner of survey statistics here, i would be surprised if anyone has published on this topic (but could be wrong) $\endgroup$ Commented Apr 6, 2016 at 2:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.