Skip to main content
added 104 characters in body; added 2 characters in body
Source Link
Not Durrett
  • 381
  • 1
  • 3
  • 5

Consider data where each observation was generated as follows.

  • We draw $Z_1,...,Z_m$ from some distribution. (Possibly they're independent or related in some other simple way.)

  • Next, based on the $Z_1,...,Z_m$, we choose a sequence $0=I_0 < I_1 < ... < I_N=m$ so that, for each $k$, (i) $I_k-I_{k-1}$ is not too small and (ii) the sample variance within $Z_{I_{k-1}},...,Z_{I_k}$$Z_{I_{k-1}+1},...,Z_{I_k}$ is small. (I am intentionally somewhat vague here - I am open to making various different assumptions along these lines.)

  • We generate the observed variables $X_1,...,X_N$ as $X_k=$ the average of $Z_{I_{k-1}+1},...,Z_{I_k}$.

For example, the hidden sequence $Z=(0.1, 0.3, 0.2, 1.3, 1.2, 0.1)$ might lead to the observed sequence $X=(0.2, 1.25, 0.1)$ [or perhaps to $X=(0.2,0.86)$ due to (i) above].

Does anyone here know whether this type of setup has been studied before, and if so, what are some keywords to search for or papers/books to look at?

Thanks in advance for any answers!

Added on Apr 21: The motivation is as follows. Think of each $Z$ sequence as SNP data from a single patient. In order to anonymize the data for a public release a procedure like the one I described above can be performed. Based on the anonymized data $X$, I want to predict survival and/or to identify SNPs that are relevant to survival.

Note that $I$, $N$, and $X$ are all functions of $Z$, so they will be different for each patient. Also note that that $I$'s are observed, i.e., I know which $Z_j$'s were averaged to produce each $X_k$.

Consider data where each observation was generated as follows.

  • We draw $Z_1,...,Z_m$ from some distribution. (Possibly they're independent or related in some other simple way.)

  • Next, based on the $Z_1,...,Z_m$, we choose a sequence $0=I_0 < I_1 < ... < I_N=m$ so that, for each $k$, (i) $I_k-I_{k-1}$ is not too small and (ii) the sample variance within $Z_{I_{k-1}},...,Z_{I_k}$ is small. (I am intentionally somewhat vague here - I am open to making various different assumptions along these lines.)

  • We generate the observed variables $X_1,...,X_N$ as $X_k=$ the average of $Z_{I_{k-1}+1},...,Z_{I_k}$.

For example, the hidden sequence $Z=(0.1, 0.3, 0.2, 1.3, 1.2, 0.1)$ might lead to the observed sequence $X=(0.2, 1.25, 0.1)$ [or perhaps to $X=(0.2,0.86)$ due to (i) above].

Does anyone here know whether this type of setup has been studied before, and if so, what are some keywords to search for or papers/books to look at?

Thanks in advance for any answers!

Added on Apr 21: The motivation is as follows. Think of each $Z$ sequence as SNP data from a single patient. In order to anonymize the data for a public release a procedure like the one I described above can be performed. Based on the anonymized data $X$, I want to predict survival and/or to identify SNPs that are relevant to survival.

Note that $I$, $N$, and $X$ are all functions of $Z$, so they will be different for each patient.

Consider data where each observation was generated as follows.

  • We draw $Z_1,...,Z_m$ from some distribution. (Possibly they're independent or related in some other simple way.)

  • Next, based on the $Z_1,...,Z_m$, we choose a sequence $0=I_0 < I_1 < ... < I_N=m$ so that, for each $k$, (i) $I_k-I_{k-1}$ is not too small and (ii) the sample variance within $Z_{I_{k-1}+1},...,Z_{I_k}$ is small. (I am intentionally somewhat vague here - I am open to making various different assumptions along these lines.)

  • We generate the observed variables $X_1,...,X_N$ as $X_k=$ the average of $Z_{I_{k-1}+1},...,Z_{I_k}$.

For example, the hidden sequence $Z=(0.1, 0.3, 0.2, 1.3, 1.2, 0.1)$ might lead to the observed sequence $X=(0.2, 1.25, 0.1)$ [or perhaps to $X=(0.2,0.86)$ due to (i) above].

Does anyone here know whether this type of setup has been studied before, and if so, what are some keywords to search for or papers/books to look at?

Thanks in advance for any answers!

Added on Apr 21: The motivation is as follows. Think of each $Z$ sequence as SNP data from a single patient. In order to anonymize the data for a public release a procedure like the one I described above can be performed. Based on the anonymized data $X$, I want to predict survival and/or to identify SNPs that are relevant to survival.

Note that $I$, $N$, and $X$ are all functions of $Z$, so they will be different for each patient. Also note that that $I$'s are observed, i.e., I know which $Z_j$'s were averaged to produce each $X_k$.

added 444 characters in body
Source Link
Not Durrett
  • 381
  • 1
  • 3
  • 5

Consider data where each observation was generated as follows.

  • We draw $Z_1,...,Z_m$ from some distribution. (Possibly they're independent or related in some other simple way.)

  • Next, based on the $Z_1,...,Z_m$, we choose a sequence $1=I_0 < I_1 < ... < I_N=m$$0=I_0 < I_1 < ... < I_N=m$ so that, for each $k$, (i) $I_k-I_{k-1}$ is not too small and (ii) the sample variance within $Z_{I_{k-1}},...,Z_{I_k}$ is small. (I am intentionally somewhat vague here - I am open to making various different assumptions along these lines.)

  • We generate the observed variables $X_1,...,X_N$ as $X_k=$ the average of $Z_{I_{k-1}},...,Z_{I_k}$$Z_{I_{k-1}+1},...,Z_{I_k}$.

For example, the hidden sequence $Z=(0.1, 0.3, 0.2, 1.3, 1.2, 0.1)$ might lead to the observed sequence $X=(0.2, 1.25, 0.1)$ [or perhaps to $X=(0.2,0.86)$ due to (i) above].

Does anyone here know whether this type of setup has been studied before, and if so, what are some keywords to search for or papers/books to look at?

Thanks in advance for any answers!

Added on Apr 21: The motivation is as follows. Think of each $Z$ sequence as SNP data from a single patient. In order to anonymize the data for a public release a procedure like the one I described above can be performed. Based on the anonymized data $X$, I want to predict survival and/or to identify SNPs that are relevant to survival.

Note that $I$, $N$, and $X$ are all functions of $Z$, so they will be different for each patient.

Consider data where each observation was generated as follows.

  • We draw $Z_1,...,Z_m$ from some distribution. (Possibly they're independent or related in some other simple way.)

  • Next, based on the $Z_1,...,Z_m$, we choose a sequence $1=I_0 < I_1 < ... < I_N=m$ so that, for each $k$, (i) $I_k-I_{k-1}$ is not too small and (ii) the sample variance within $Z_{I_{k-1}},...,Z_{I_k}$ is small. (I am intentionally somewhat vague here - I am open to making various different assumptions along these lines.)

  • We generate the observed variables $X_1,...,X_N$ as $X_k=$ the average of $Z_{I_{k-1}},...,Z_{I_k}$.

For example, the hidden sequence $Z=(0.1, 0.3, 0.2, 1.3, 1.2, 0.1)$ might lead to the observed sequence $X=(0.2, 1.25, 0.1)$ [or perhaps to $X=(0.2,0.86)$ due to (i) above].

Does anyone here know whether this type of setup has been studied before, and if so, what are some keywords to search for or papers/books to look at?

Thanks in advance for any answers!

Consider data where each observation was generated as follows.

  • We draw $Z_1,...,Z_m$ from some distribution. (Possibly they're independent or related in some other simple way.)

  • Next, based on the $Z_1,...,Z_m$, we choose a sequence $0=I_0 < I_1 < ... < I_N=m$ so that, for each $k$, (i) $I_k-I_{k-1}$ is not too small and (ii) the sample variance within $Z_{I_{k-1}},...,Z_{I_k}$ is small. (I am intentionally somewhat vague here - I am open to making various different assumptions along these lines.)

  • We generate the observed variables $X_1,...,X_N$ as $X_k=$ the average of $Z_{I_{k-1}+1},...,Z_{I_k}$.

For example, the hidden sequence $Z=(0.1, 0.3, 0.2, 1.3, 1.2, 0.1)$ might lead to the observed sequence $X=(0.2, 1.25, 0.1)$ [or perhaps to $X=(0.2,0.86)$ due to (i) above].

Does anyone here know whether this type of setup has been studied before, and if so, what are some keywords to search for or papers/books to look at?

Thanks in advance for any answers!

Added on Apr 21: The motivation is as follows. Think of each $Z$ sequence as SNP data from a single patient. In order to anonymize the data for a public release a procedure like the one I described above can be performed. Based on the anonymized data $X$, I want to predict survival and/or to identify SNPs that are relevant to survival.

Note that $I$, $N$, and $X$ are all functions of $Z$, so they will be different for each patient.

Tweeted twitter.com/#!/StackStats/status/60993502695264256
Source Link
Not Durrett
  • 381
  • 1
  • 3
  • 5

Averages of random subsets of variables

Consider data where each observation was generated as follows.

  • We draw $Z_1,...,Z_m$ from some distribution. (Possibly they're independent or related in some other simple way.)

  • Next, based on the $Z_1,...,Z_m$, we choose a sequence $1=I_0 < I_1 < ... < I_N=m$ so that, for each $k$, (i) $I_k-I_{k-1}$ is not too small and (ii) the sample variance within $Z_{I_{k-1}},...,Z_{I_k}$ is small. (I am intentionally somewhat vague here - I am open to making various different assumptions along these lines.)

  • We generate the observed variables $X_1,...,X_N$ as $X_k=$ the average of $Z_{I_{k-1}},...,Z_{I_k}$.

For example, the hidden sequence $Z=(0.1, 0.3, 0.2, 1.3, 1.2, 0.1)$ might lead to the observed sequence $X=(0.2, 1.25, 0.1)$ [or perhaps to $X=(0.2,0.86)$ due to (i) above].

Does anyone here know whether this type of setup has been studied before, and if so, what are some keywords to search for or papers/books to look at?

Thanks in advance for any answers!