Averages of random subsets of variables

Question

Consider data where each observation was generated as follows.

• We draw $Z_1,...,Z_m$ from some distribution. (Possibly they're independent or related in some other simple way.)

• Next, based on the $Z_1,...,Z_m$, we choose a sequence $0=I_0 < I_1 < ... < I_N=m$ so that, for each $k$, (i) $I_k-I_{k-1}$ is not too small and (ii) the sample variance within $Z_{I_{k-1}+1},...,Z_{I_k}$ is small. (I am intentionally somewhat vague here - I am open to making various different assumptions along these lines.)

• We generate the observed variables $X_1,...,X_N$ as $X_k=$ the average of $Z_{I_{k-1}+1},...,Z_{I_k}$.

For example, the hidden sequence $Z=(0.1, 0.3, 0.2, 1.3, 1.2, 0.1)$ might lead to the observed sequence $X=(0.2, 1.25, 0.1)$ [or perhaps to $X=(0.2,0.86)$ due to (i) above].

Does anyone here know whether this type of setup has been studied before, and if so, what are some keywords to search for or papers/books to look at?

Thanks in advance for any answers!

Added on Apr 21: The motivation is as follows. Think of each $Z$ sequence as SNP data from a single patient. In order to anonymize the data for a public release a procedure like the one I described above can be performed. Based on the anonymized data $X$, I want to predict survival and/or to identify SNPs that are relevant to survival.

Note that $I$, $N$, and $X$ are all functions of $Z$, so they will be different for each patient. Also note that that $I$'s are observed, i.e., I know which $Z_j$'s were averaged to produce each $X_k$.

Answer 1

You could apply linear regression with regularization (Lasso) to solve this problem. The idea would be to fit the data with piece-wise constant functions adding a penalty for every jump that occurs. The objective you have to minimize is

$x^* = \arg\min_{x\in\mathbb{R}^m} \|z - x\|_2^2 + \lambda \|\nabla x\|_1$,

where $(\nabla x)_i = x_{i-1} - x_{i}$ is the (backward) difference operator on the grid. The parameter $\lambda$ controls the tradeoff between small- vs. large intervals and low vs. high variance within the intervals.

From the solution $x^*$ you can then reconstruct the intervals $I$.

Answer 2

Your problem would fall under the category of "missing data." Ultimately, one way or another you are going to have to infer the hidden variables $Z$. This can be done using the Expectation-Maximization Algorithm.