Predicting dependent variables where independent variables are samples with differing sample size

Question

Is there a standard method of dealing with independent variables, where they are samples from a known distribution, and the sample sizes differ from sample to sample?

I'll give an example of what I mean. Suppose your sample consists in $n$ individuals, who have been given 3 different types of test. Let $Z_{i,j}$ be number of successes for person $i$, on test $j$; $n_{i,j}$ is the number of times person $i$ takes test $j$ (the sample size). We further assume that $Z_{i,j} \sim B(n_{i,j}, p_{i_j})$, where $p_{i,j}$ represents the ability of person $i$, at test $j$. The different individuals have different abilities at the test.

We want to use the sample to construct a model that let us use the results of test 1 and 2 for an individual to predict their result on test 3. We could fit a generalized linear model, for example, where $Z_{i,3}$ is the independent variable, and $\frac{Z_{i,1}}{n_{i,1}}$ and $\frac{Z_{i,2}}{n_{i,2}}$ are dependent variables, but we wouldn't take into account the differing sample sizes. Ideally, we want to construct a model that accounts for the relationship between an individual's ability on the different tests ($p_{i,1}, p_{i,2},p_{i,3}$), and use that to predict his ability on test 3, given his ability on tests 1 and 2, but we only know the results of the tests, the $Z_{i,j}$'s.

ADDITION: Suppose we have a model of the relationship between each of the $p_{i,\cdot}$'s, for example $p_{i,3}=f(\beta_0+\beta_1g(p_{i,1})+\beta_2g(p_{i,2}))$, where $f$ and $g$ are, say, logit functions. How would I obtain those parameters? Is there a standard algorithm for this, or a function in R? Is maximum likelihood an appropriate method of fitting?

Also, when fitting the model, what pitfalls are there? Are there metrics or tests for comparing different models, or determining goodness-of-fit? Is there a term for this form of modelling?

Answer 1

This question seems to be describing a situation where you have three tests on three different time points. A typical way of addressing such a  a situation would be to use a multidimensional item response model, where the different time points are the dimensions (that is assuming that you are interested in individual variation, otherwise the model can be simplified considerably).

However, this model requires you to have access to the answers to the individual tasks that comprise each item AND to have "anchor items" (i.e. items that are present in more than one time point). If I am understanding the setup in your questions, your tests at the three time points are comprised by repeated trials of the same questions, which would mean that you would need to use the same task across the three times  in order to be able to "link" the results of the three tests into a common scale.

In any case, the multidimensional model would not have problems with the different number of items on each time point (notice that to run this kind of model the data is usually reshaped into "long" format, and your response variable are the actual responses to the questions). I hope this help at least as a starting point.

This is the model that came to mind when reading the question:

Where:

$p$ stands for person, $i$ stands for item, and $t$ stands for item and takes values 0, 1 and 2.

$d_{it_1}$ is a dummy variable that equals 1 if the item is present on time point 1 and $d_{it_2}$ is equivalent for time point 2.

The model would be:

$$Logit[ Pr( x_{pi}) = 1 ] = \beta_0 + \zeta_{0p} + \beta_1 d_{it_1} + \zeta_{1p} d_{it_1} + \beta_2 d_{it_2} + \zeta_{2p} d_{it_2} - \delta_i$$

$$\mathbf{\zeta} \sim MVN(0,\mathbf{\Sigma})$$

Then:

$\beta_0$ corresponds to the average proficiency at time 0.

$\beta_1$ corresponds to the difference in proficiency between time 1 and time 0.

$\beta_2$ corresponds to the difference in proficiency between time 2 and time 0.

$\zeta_{0p}$ is the random effect accounting for individual variation of proficiency at time 0.

$\zeta_{1p}$ is the random effect accounting for individual variation of proficiency at time 1.

$\zeta_{2p}$ is the random effect accounting for individual variation of proficiency at time 2.

$\delta_i$ correspond to the item difficulty which is assumed to remain constant over time.