Let there be observed data points $X = {X_1.. X_n .. X_N}$, where each $X_n \in R^D$. Lets assume these are distributed as a Gaussian $X \cong \mathcal N(\mu,\Sigma)$. Let us also assume that the mean has a Normal prior and covariance has Inverse Wishart prior. Since these are conjugate priors, given observed data $X$, we can draw posterior samples for mean and covariance using standard posterior update equations. What we have so far is just the usual Bayesian stuff.
Let the data points be formed from a union of two sets of data $X = {X^+} \cup {X^-}$. The data is still distributed as Gaussian. However lets assume that while drawing posteriors for mean and covariance, we have to give more weight to data points in ${X^+}$ than ${X^-}$, with the weight itself being some (hyper)parameter.
Note that we are NOT talking about mixture of gaussians. We dont say there are 2 mixtures and the data might have arised from one of these mixtures. We say the data still arises from ONE gaussian, with the mean and covariance primarily according to ${X^+}$, but perhaps altered(shifted?) by some proportion because of ${X^-}$ influence.
I am not sure how i can write the distribution for such a thing. I initially thought i can write $X \cong \mathcal N(A\mu^+ + B\mu^-,A\Sigma^+ + B\Sigma^-)$. We then draw $\mu^+, \Sigma^+$ from ${X^+}$ and similarly $\mu^-, \Sigma^-$ from ${X^-}$ by usual methods. We treat $A$ and $B$ as some (hyper)parameters. If i write like this, what is my assumption of the data. Am i implying $X = {X^+} + {X^-}$ rather than a union? Or the implication is too non-sensical to even worth a description?
Why would i want to do this - thats a lengthy explanation and would distract the specific question. The specific question is - "I have data partitioned into two sets. The data arises from a gaussian distribution. I want to draw posterior samples for the gaussian parameters according to one of the set. However the other set also has (marginal) influence on the parameters. How can I re-parametrize the distribution to account for this?"
Ps: Any approximate re-parametrization is fine. I've used the word "sampling", hopefully that wouldn't distract the specific question to ideas such as lets reject samples based on some function. Any references to the literature that does something similar is also welcome. Thanks in advance for suggestions.
