2
$\begingroup$

I have a very simple data set with just one real valued feature ($x_i$) and a real valued target ($y_i$).

My model assumes that the targets depend on the feature in a very simple way: for the features that are smaller than some unknown split value ($x_i < x_{split}$), the targets come form one distribution, otherwise ($x_i > x_{split}$) they come from another distribution. I assume that these two distributions have different means and, in the end, I am interested only in these means ($\mu_1$ and $\mu_2$) (I do not care about their dispersion, skewness, kurtosis and so on).

To summarize, I would like to extract from my data set three parameters (my model): (1) location of the split: $x_{split}$, (2) mean of the first distribution: $\mu_1$, (3) mean of the second distribution: $\mu_2$.


My first approach was to find those parameters by minimization of squared deviations (this is the standard approach used by decision trees and other regressors):

$ E = n_1 \cdot \sigma_1^2 + n_2 \cdot \sigma_2^2 $

where $n_1$ and $n_2$ are number of the points in each of the two splits ($x < x_{split}$ and $x > x_{split}$) and $\sigma_1$ and $\sigma_2$ are squared standard deviations of the targets from their respective means within the split:

$ \sigma_1^2 = \frac{1}{n_1} \sum_{i=1}^{n_1} (y_i - \mu_1)^2 $

$ \sigma_2^2 = \frac{1}{n_2} \sum_{i=n_1+1}^{n_1 + n_2} (y_i - \mu_2)^2 $

Here I assume that the targets are ordered by the feature.


It was fine until I decided to try to maximize likelihood. I have assumed the following model. As before, there is a split somewhere (we want to find where) and on the two sides of the split we have two different normal distributions: $N(\mu_1, \sigma_1)$ and $N(\mu_2, \sigma_2)$. I wanted to find the model which maximizes the likelihood.

By simple analytical derivations I found that to find the location of the split I need to minimize the following measure:

$ E_{new} = n_1 \cdot \ln \sigma_1 + n_2 \cdot \ln \sigma_2 $

It is obviously different from the old measure:

$ E_{old} = n_1 \cdot \sigma_1^2 + n_2 \cdot \sigma_2^2 $


So, since I have two inconsistent measures I am not sure anymore what one to use. Both give me a location of the split as well as means on the two part of the split but location of the splits are different. Which one is more correct (whatever it means)?


Sketch of Derivation:

First I prove a thing that might be trivial for other and is not even need to be proved: For a given location of the split the average and dispersions of the two normal distributions should be equal to the observed sample means and dispersions. On the second step I just insert the observed $\mu_i$ and $\sigma_i$ into the formula of log-likelihood assuming normal distributions.

Derivation 1:

enter image description here

Derivation 2:

enter image description here

$\endgroup$
10
  • $\begingroup$ maybe you could present your working. your ML metric doesn't seem plausible... n x ln(sigma) seems a meaningless quantity $\endgroup$
    – seanv507
    Jun 9, 2022 at 12:15
  • $\begingroup$ I have added my derivations. $\endgroup$
    – Roman
    Jun 9, 2022 at 12:30
  • 1
    $\begingroup$ Please type out your derivations. $\endgroup$
    – Dave
    Jun 9, 2022 at 13:27
  • 1
    $\begingroup$ @seanv507 I guess you did not get what I am talking about. Split is done by the feature and normal distributions are assumed for targets. It is like, if age is below 40 years, cholesterol levels come from one distribution, it age is above 40 years, the cholesterol levels come from another distribution. $\endgroup$
    – Roman
    Jun 9, 2022 at 19:27
  • 1
    $\begingroup$ You are right I thought you were clustering the feature. Have deleted my comments.You can rewtite your second objective as the geometric mean. so then the objectives are more similar - geometric vs arithmetic mean of variances $\endgroup$
    – seanv507
    Jun 10, 2022 at 7:00

1 Answer 1

0
$\begingroup$

Notice that the mean squared error of the first approach can also arise from likelihood maximization, but for a model that assumes the same variance for the two normal distributions. Namely if you assume that the distributions are $N(\mu_1,\sigma_0)$ and $N(\mu_2,\sigma_0)$, then by simple calculation you can find that the maximum likelihood for a given split is

$$-2\mathcal {\hat L} = \frac{1}{\sigma_0^2}(n_1 \cdot \sigma_1^2 + n_2 \cdot \sigma_2^2) $$

Minimizing this is equivalent to minimizing your measure $E_{old}$ since $\sigma_0$ is just a constant, and it doesn't matter which value we assume $\sigma_0$ actually takes.

So the difference between the two approaches can be seen as having different assumptions on the underlying distributions. To further see it consider a case where the data is divided to two groups with the same mean $\mu$ but different variances. in this case we have

$$ n_1\sigma_1^2 + n_2\sigma_2^2 = \sum_i (y_i - \mu)^2 $$

which means that this quantity will be the same for all splits - it is not sensitive at all to the difference in variances. On the other hand $n_1 \log \sigma_1 + n_2\log \sigma_2$ will prefer the split with different variances, which we can see e.g. using Jensen's inequality:

$$ n_1 \log \sigma_1 + n_2\log \sigma_2 \le n \log( \frac{n_1\sigma_1^2 + n_2\sigma_2^2}{n} ).$$

(where $n=n_1+n_2$).

Having the variances as free parameters therefore means that the likelihood will prefer splits that generate distributions as "different" as possible, where this difference can be either in mean or variance. A model that assumes the same variance for both distributions will only consider the difference in means - which might be more in line with what you actually want.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.