# Modeling outliers in maximum likelihood estimation with gradient descent

Consider a set of 3D points $$X = \{x_1, x_2, ...x_n\}$$ with $$x_i\in\mathbb{R}^3$$ on which we want to fit an arbitrary probability distribution. The distribution we want to fit models some geometrical shape, like a sphere, a torus or a cylinder, but in principle this detail is irrelevant for my question. Just assume we can evaluate the PDF of the distribution.

Assuming there is no closed form solution for the maximum likelihood estimate of the parameters of the distribution, we can use an iterative gradient descent method to minimize the negative log likelihood:

$$\mathcal L = \sum_{x \in X} -\log(P(x \, | \,\theta))$$

where $$P(x \, | \,\theta)$$ is the probability density function and $$\theta$$ is the vector of parameters to optimize.

However, imagine that the points we are trying to fit include some blobs of outliers, which cannot be modelled with our distribution. Inspired by this paper [1], I thought one way to handle the outliers could be to include a constant hyper-parameter $$\Omega$$ in the loss function which gives each point a minimum probability:

$$\mathcal L = \sum_{x \in X} -\log(P(x \, | \,\theta) + \Omega)$$

This way, the loss won't go to infinity as soon as one outlier point is not well fitted by the distribution and has $$P(x \, | \,\theta) \approx 0.$$ Having tried this in practice, I can confirm this allows fitting the object of interest while ignoring the blobs of outlier points, even in cases where the number of outlier points outnumbers the inliers by far.

Moreover, I understand this is somewhat equivalent to fitting a mixture model with two components: 1) our arbitrary distribution with PDF $$P(x \, | \,\theta)$$ and 2) a uniform PDF with constant value $$\Omega$$. In this regard, would it make sense to introduce two more parameters into the optimization problem to model the mixture weights? Or is this unnecesary, given that one of the mixture components has no parameters to optimize.

$$\mathcal L = \sum_{x \in X} -\log(w_1 \, P(x \, | \,\theta) + w_2 \, \Omega) \\ s.t. ~~ w_1 + w_2 = 1$$

In addition, are there other ways to make the fitting robust to outliers in a setup like this?

[1] https://arxiv.org/abs/2303.15440 (see Eq. 7)

The parameters have no effect on the minimisation. The effect is already regulated with the single parameter $$\Omega$$

$$\qquad \hat{\theta} = \min_\theta -\mathcal L(\theta)$$

$$\qquad \hat{\theta} = \min_\theta \sum_{x \in X} -\log(w_1 \, P(x \, | \,\theta) + w_2 \, \Omega)$$

using $$w_2 = 1-w_1$$

$$\qquad \hat{\theta} = \min_\theta \sum_{x \in X} -\log(w_1 \, P(x \, | \,\theta) + (1-w_1) \, \Omega)$$

rearranging terms

$$\qquad \hat{\theta} = \min_\theta \sum_{x \in X} -\log( \, P(x \, | \,\theta) + \frac{1-w_1}{w_1} \, \Omega) -\log(w_1)$$

We can drop the term $$\log(w_1)$$ which only shifts the likelihood with a constant and has no effect on the minimisation

$$\qquad \hat{\theta} = \min_\theta \sum_{x \in X} -\log( \, P(x \, | \,\theta) + \frac{1-w_1}{w_1} \, \Omega)$$

$$\frac{1-w_1}{w_1}\Omega$$ can be combined into a single constant

$$\qquad \hat{\theta} = \min_\theta \sum_{x \in X} -\log( \, P(x \, | \,\theta) + \Omega^\prime)$$

In addition, are there other ways to make the fitting robust to outliers in a setup like this?

With the current method you assume that an outlier is distributed homogeneously. If you know more properties about outliers, and if you know more about the true distribution (e.g. potential correlation between measurements), then you can incorporate this into the likelihood function.

In addition you may have some cost function that you want to minimize, this you may add in a cross validation step to optimize the model hyper-parameters.

• Thanks for your answer. However, note that in the original formulation $\omega$ is not an optimizable parameter, but a fixed hyper-parameter. From your analysis, I take that introducing the optimizable mixture weights while keeping $\omega$ fixed is equivalent to making $\omega$ optimizable. Is that right? May 26, 2023 at 9:29
• @DanielLópez Yes, 'making your constant $\Omega$ a variable $\Omega$' versus 'multiplying it with a variable', that has the same effect. To me it would make more sense to do the first and not the second. Why keep this constant when you can combine it with that multiplicative variable? May 26, 2023 at 9:37
• I think if you just make $\Omega$ an unbounded variable, it will just blow up to infinity, because you can monotonically reduce the loss by increasing $\Omega$. The mixture weights, which are constrained to $w_1 + w_2 = 1$, introduce a trade-off: you can increase the probability contributed by the uniform component of the mixture, but at the cost of reducing the contribution of the $P(x|\theta)$ component. I think you neglected that trade-off when you dropped the $-log(w_1)$ term. Are you sure you can do that when you are actually also tuning $w_1$?. May 26, 2023 at 9:50
• Yes, that is true if you optimize $\Omega$ as a model parameter along with $\theta$ to maximize the likelihood. In that case the step where $-log(w_1)$ is dropped should indeed be skipped. In that case you can drop the $\Omega$ term (set it equal to $\Omega = 1$), and use only a single parameter $w_1$. May 26, 2023 at 10:03
• Okay, that makes it clear. Feel free to edit the response to summarize our discussion, which I think is relevant to make the answer more complete. Otherwise I will accept the answer as it is. Thanks a lot! May 26, 2023 at 10:47