# What information, other than directly observed data, can a likelihood encode?

## Background

Typically, the likelihood function is defined as the probability of observed data under an assumed model. So something like: $$L(\theta | {\bf x}, {\mathcal M}) = p_{\mathcal M}({\bf x}|\theta)$$

Here, the likelihood function explicitly depends on the data $$\bf x$$.

If observations are censored, the problem is framed in a clever way where, for example, if we observe the random variable $$Y$$ up until a certain value $$l$$ and $$y_l$$ if $$Y \geq y_l$$, the observation itself is treated as a random variable with a Bernoulli distribution, so the problem of writing down a likelihood for the model again becomes a well defined and solvable problem.

My question:

What if you observe something else that's not data? For example, lets say $$f(x)$$ and $$g(x)$$ are functions. My model for both $$f$$ and $$g$$ is a Gaussian Process (as a GP is a model on functions).

If I had observed some data for $$f$$ and $$g$$, I could easily write down a likelihood: $$log \; L(\theta;{\bf y}_f, {\bf y}_g) = c - \frac{1}{2} y_f^T\Sigma^{-1}_{x, \theta}y_f - \frac{1}{2} y_g^T\Sigma^{-1}_{x, \theta}y_g$$

## Problem

Now, what if one "observes" that $$g = sin(f)$$? Can this information be encoded within the likelihood? Would it be correct to say that:

$$log \; L(\theta;{\bf y}_f, {\bf y}_g) = c - \frac{1}{2} y_f^T\Sigma^{-1}_{x, \theta}y_f - \frac{1}{2} sin(y_f)^T\Sigma^{-1}_{x, \theta}sin(y_f)$$

Ideally, it's true that f really should be treated as a transformation of a random variable, but sometimes the transformation can be non-monotonic and here, it was pointed out that a reasonable distribution cannot be 'assigned' to a non-monotonic transformation of a RV knowing nothing about what the transformation actually is.

I understand that changing the likelihood in this manner changes the probability law, and it enforces a joint distribution that might not actually be a distribution, but why would this be wrong? I've sincerely "observed" that all my $$y_g$$s were just $$sin(y_f)$$s!

If this can't be done, then how could one account for this information?

Edit in the light of @dwcoder's answer:

I had briefly considered these arguments while writing the likelihood that I stated above; there are two main reasons (within the context of this answer) for why I wrote the likelihood the way I did:

1. (The excuse) For my specific example, the likelihood has two components to it, which are the MVN densities of the two GPs. If we add a third contribution, such as the likelihood of the statement $$y_g = sin(y_f) + \epsilon$$ and let $$\epsilon \rightarrow 0$$ a.s., my guess is that it enforces the likelihood that I wrote above (because $$y_g$$ and $$y_f$$ are free parameters and the additional term forces $$y_g$$ to be close to $$sin(y_f)$$).

2. (The reason) When it comes to model building, I had always thought that we first build a model with some assumptions, do the experiment, and then feed in the results. What if before the experiment I do not actually know what the relationship between $$y_f$$ and $$y_g$$ will be, and this is a result of the experiment? I'm not really trying to encode assumptions directly into the likelihood - I'm trying to avoid encoding the results into my model, before I see them.

... which is essentially my question - the standard frameworks such as the one in Edwards have well formed arguments, but sometimes, there is more information in an experiment over and above the observed data points corresponding to the model's random variables.

At what point does "information" stop being something you can plug into the likelihood and become something for which you need special care to put into a model? I still don't understand why it's wrong to state the likelihood the way I did.

As for building a model that in some way captures the dependance between $$y_g$$ and $$y_f$$ - I don't see why this is needed. Two independent samples/draws from RBF-GPs, when plotted against each other, have very nice & smooth 'joint relationships'. For example, here's a simulation: The two components are independent on average, but each set of samples has a great amount of interdependence. How would one account for this given that the independent GPs model is the "true" model?

Edit 2:

Another thing that comes to mind is that, a likelihood doesn't have to integrate to anything - it simply reflects the probability of observing something (or atleast, it's proportional to it). Hence, evaluating the above density at $$y_f$$ and $$sin(y_f)$$ doesn't lose the interpretation of the likelihood.

It depends on what you mean by "encode". The likelihood is defined in Likelihood (Edwards, 1984) as a function in the parameters that is proportional to the probability of observing the data: $$L(\theta; x , \mathcal{M}) \propto p_{\mathcal{M}}(x|\theta)$$ and this is the fundamental definition from which the likelihood given in statistics textbooks usually comes. Although they seldom state this explicitly, they usually define it in terms of the density functions: $$L(\theta; x , \mathcal{M}) = p_{\mathcal{M}}(x|\theta)$$ which isn't wrong, it just ignores some information.

Now for your question, what information can the likelihood "encode", other than that in the data? Since the likelihood is constructed using the probability, it will contain all the assumptions you make about the data.

If you assume $$x_i \sim \text{Normal}(\mu, \sigma^2), \text{ i.i.d}$$ for $$i = 1 \ldots n$$, you will get a likelihood that contains your assumption about the Normal distribution $$L(\theta| x ,\mathcal{M}) = \prod_{i=1}^{n}{ f_{\text{N}(\mu, \sigma)}(x_i) }$$ where $$f_{\text{N}(\mu, \sigma)}(x_i)$$ is the pdf of the normal distribution. What does this "encode"? Well, we constructed the likelihood using our assumptions of normality, and independence.

If you want to "encode" some information about a sinusoid, you have to decide what this means for your probability assumptions. Perhaps you observe $$x_t$$ as a time series and you want it to follow a sinusoidal shape with an error:

$$x_t \sim \text{Normal}(\theta_1 sin(t + \theta_2), \sigma^2), \text{ i.i.d}$$ And then the likelihood becomes: $$L(\theta| x , \mathcal{M}) = \prod_{t=1}^{n}{ f_{\text{N}( \theta_1 sin(t) + \theta_2 , \sigma^2)}(x_t) }$$ I think this is what you are alluding to. Perhaps you mean: $$y_g = sin(y_f) + \varepsilon, \;\; \varepsilon \sim N(0, \sigma^2)$$ meaning $$y_g \sim \text{Normal}( sin(y_f) , \sigma^2) \text{.}$$ Either way, don't try to construct the likelihood in such a way that it "encodes" your assumptions. Write out your assumptions probabilistically, and then derive the likelihood from that.

The difficult bit is expressing your modelling assumptions probabilistically. You mention that some transformations don't lead to well definited probability distributions, and then you can't write down a likelihood. But you are observing some relationship between two GPs. So you will have to think of innovative ways to express this relationship in something that can be modeled. I will not attempt to do that here, since it might require a whole paper.

• Thanks for the answer! I've added an edit to my post explaining some of the aspects of my question. – InfProbSciX Oct 4 '18 at 18:54

I'm wrong, I was being ridiculous.

As @dwcoder suggested, expressing any relationship between two random variables tampers whatever joint distribution they had. The new observed joint distribution must replace the likelihood.

Intuitively, if the likelihood is the probability of seeing some data, seeing the data doesn't change the structure of the joint relationship between variables. Seeing literally anything else does change the relationship between variables and not updating the likelihood in an appropriate fashion leads to expressing the wrong joint probability.

As a simple example, if $$X, Y \sim D. Uniform(\{-2, -1, 0, 1, 2 \})$$ and we observe that $$Y = abs(X)$$, the joint probability changes from $$1/25$$ for any $$x, y$$ to $$1/5$$.

For GPs, I guess that this is more complicated, but simply replacing $$y_g$$ with $$sin(y_f)$$ would underestimate the probability (the likelihood) as observing that $$g = sin(f)$$ has shrunk the domain of $$g$$ - some paths are now impossible hence others must have become more probable.