If we assume that we have a set of N data points given as $\textbf{X}$ and corresponding targets vectors $\textbf{T}$, where both represents matrices in this case. For an i.i.d we could write the likelihood (by using vectorization), through:

$$p(\textbf{T}\vert f,\textbf{X}) = \prod_{i=1}^{N}p(\textbf{t}_i\vert f,\textbf{X})$$ Where I've denoted f as an linear estimator of $\textbf{t}_i$. Now I want to write the likelihood for a not i.i.d, I thought it might look like:

$$p(\textbf{T}\vert f,\textbf{X})= \prod_{i=1}^{N}p(\textbf{t}_i\vert \textbf{t}_{i+1},...,\textbf{t}_N,f,\textbf{X})$$

But this doesn't seem right, it seems like I've forgotten something, and if that's not the case, it feels like I'm calculating the same points over and over. How can I fix this ?

Personally I think it should look like:

$$p(\textbf{T}\vert f,\textbf{X}=p(\textbf{t}_1\vert f,\textbf{X})*p(\textbf{t}_1\vert \textbf{t}_2,...,\textbf{t}_N,f,\textbf{X}) * p(\textbf{t}_2\vert f, \textbf{X})*p(\textbf{t}_2\vert \textbf{t_1},...,),....p(\textbf{t}_N$$


You can factorise a joint distribution (the likelihood is the joint density of the data written as a function of unknown parameters) without assuming anything about how the variables depend on each other by starting with the marginal distribution of the first (or any arbitrary) variable and gradually enlarging the conditioning set to include previous variables:

$$p(t_1, t_2, t_3, ..., t_N) = p(t_1)p(t_2|t_1)p(t_3|t_1, t_2) ... p(t_N| t_1,..., t_{N-1})$$

Why? For just two variables, it follows from the definition of conditional probability: $$\Pr(A,B) = \Pr(B) \cdot \Pr(A|B)$$

  • $\begingroup$ Perhaps you can clarify something that I might have misunderstood, should I interpret it as $\textbf{T}$ is given in my case, therefore assuming that I have knowledge of these, or should I base my calculations around the fact that I dont have $\textbf{T}$, rather I calculate the likelihood for $\textbf{t}_1$ in the first place, and then extend this argument when adding more points ? $\endgroup$ – A.Maine Nov 15 '18 at 10:35
  • $\begingroup$ perhaps I should reformulate, is this a correct interpretation? $$p(t_1, t_2,...,t_N) = p(t_1 \vert f,\bm{X})*p(t_2\vert t_1,f, X)*...*p(t_n \vert t_1,....,t_{N-1} \vert f, X)$$ $\endgroup$ – A.Maine Nov 15 '18 at 10:50
  • $\begingroup$ If you add $f$ and $X$ to the conditioning set on the left hand side, then yes. The factorisation formula is fully general. By the way, I'm not sure what's what in your notation. Liklihoods usually refer to the joint density or mass of the data ($x$) conditional on (if Bayesian) or written as a function of (if frequentist) the parameter vector ($\theta$) one is trying to estimate, something like $p(x_i, .x_n| \theta)$ $\endgroup$ – CloseToC Nov 15 '18 at 10:55
  • $\begingroup$ Thanks a lot for the answer, I can't upvote you but I did accept the answer! $\endgroup$ – A.Maine Nov 15 '18 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.