3
$\begingroup$

In linear or logistic regression, we have the following (adapted from Foundations of machine learning.):

As in all supervised learning problems, the learner $\mathcal{A}$ receives a labeled sample dataset $\mathcal{S}$ containing $N$ i.i.d. samples $\left(\mathbf{x}^{(n)}, y^{(n)}\right)$ drawn from $\mathbb{P}_{\mathcal{D}}$:

$$ \mathcal{S} = \left\{\left(\mathbf{x}^{(1)}, y^{(1)}\right), \left(\mathbf{x}^{(2)}, y^{(2)}\right), \ldots, \left(\mathbf{x}^{(N)}, y^{(N)}\right)\right\} \subset \mathbb{R}^{D} \quad \overset{\small{\text{i.i.d.}}}{\sim} \quad \mathbb{P}_{\mathcal{D}}\left(\mathcal{X}, \mathcal{Y} ; \boldsymbol{\beta}\right) $$


I am used to the iid assumption in machine learning, but in the case of conditional maximum likelihood, I have the following question.

To use maximum likelihood for linear/logistic regression, it is required to have $y \mid x$ to be independent, in other words, $y$ is conditionally independent of $x$. The question is, do we need the strong iid assumption mentioned above for us to invoke MLE?

$\endgroup$
6
  • $\begingroup$ I do not particularly like your notation, neither the $\left(\mathbf{x}^{(n)}, y^{(n)}\right)$ nor the $y \mid x$. In essence the independence you want is $p\left(\left(\mathbf{x}^{(i)}, y^{(i)}\right), \left(\mathbf{x}^{(j)}, y^{(j)}\right)\right)= p\left(\left(\mathbf{x}^{(i)}, y^{(i)}\right)\right)\,p\left( \left(\mathbf{x}^{(j)}, y^{(j)}\right)\right)$ when $i \not=j$ and similarly for more, whether the random variables are the pairs $\left(\mathbf{x}^{(i)}, y^{(i)}\right)$ or just the $y^{(i)}$ with the $\mathbf{x}^{(i)}$ being fixed. $\endgroup$
    – Henry
    Mar 17, 2023 at 8:12
  • $\begingroup$ What do you mean by "conditional independence" here? Independent of what? For conditional independence, you need 3+ terms. $\endgroup$
    – Tim
    Mar 17, 2023 at 8:47
  • $\begingroup$ I meant that each pair of $(y^{(i)} \mid \mathbf{x}^{(i)})$ and $(y^{(j)} \mid \mathbf{x}^{(j)})$ are independent. $\endgroup$
    – nan
    Mar 17, 2023 at 9:47
  • $\begingroup$ A problem is that $(y^{(i)} \mid \mathbf{x}^{(i)})$ is not an event or a random variable in the original probability space (unless $\mathbf{x}^{(i)}$ is fixed in advance, in which case it is simply $y^{(i)}$) $\endgroup$
    – Henry
    Mar 17, 2023 at 10:32
  • $\begingroup$ Definitely do not need to assume the X's are iid, or even random. You can use maximum likelihood logistic regression in the case of designed experiments, where the X's are set by the researcher. However, one place where the (x,y) vectors are assumed iid is in the case where you bootstrap the (x,y) data vectors. These comments apply equally to all regression, not just logistic. $\endgroup$ Mar 17, 2023 at 12:32

1 Answer 1

7
$\begingroup$

If you are asking about the i.i.d. assumption in machine learning in general, we already have that question answered in the On the importance of the i.i.d. assumption in statistical learning question.

As about maximum likelihood, notice that the likelihood function is often written as

$$ \prod_{i=1}^N p(x_i | \theta) $$

where $p(x_i | \theta)$ is probability density or mass function for the point $x_i$ parameterized by $\theta$. We are multiplying because we are making the independence assumption; otherwise the joint distribution would not be a product of the individual distributions. Moreover, $p(\cdot | \theta)$ are all the same, so they are "identical", and hence we are talking about the i.i.d. assumption. This does not mean that every likelihood function would assume independence, but that is often the case. The identical distributions assumption also is not necessary, e.g. you can have a mixture model (e.g. clustering), where you assume that individual samples come from different distributions, together forming a mixture.

Notice that with maximum likelihood we are directly making such assumptions. If you are fitting a decision tree or $k$NN you are not maximizing any likelihood, the algorithms do not explicitly assume any probability distribution, so you are also not explicitly making such a assumption. It still is the case, however, that you are assuming that your data is "all alike" (so a kind of i.i.d. or exchangeability): for example, you wouldn't mix data from completely different domains (say, ice-cream sales, size of brain tumors, and speed of Formula 1 cars) together and expect it to return reasonable predictions.

As for logistic regression, that is discussed in the Is there i.i.d. assumption on logistic regression? thread.

It would be a tautology, but the assumptions that you made need to hold. If your model assumes that the samples are independent, then you need the independence assumption.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.