Use of Change of Variables (in probability distributions) in Machine Learning

Question

I am learning about machine learning from a probabilistic perspective via Kevin Murphy's so far fantastic Textbook (2021) Machine Learning - Probabilistic Machine Learning - An Introduction. I'm in the beginning, where he explains important concepts in probability and statistics used throughout the book.

Currently, I am learning about the change of variables - how to find the new Probability Distribution Function after transforming your old probability distribution by some monotonic function. It's quite fascinating and took me an embarrassingly long time to understand, but I was wondering:

Does anyone know any concrete scenarios where an understanding of this, or this method directly, is used in machine learning? Just out of curiosity.

Thanks! A

Answer 1

I'll give a fairly simple example, but this should give you an idea for how the idea is useful in other scenarios.

Suppose you fit a linear regression to some data which is strictly positive. I.e. $Y > 0$ (e.g. $Y=\text{house prices in your area}$ $x=\text{square footage of house}$). Fitting a linear regression might give us predictions which are negative. Thus a log-transform of $Y$ will allow us to construct models on an unconstrained space. So we would fit

$$ \log (Y) = \beta_0 + \beta_1 x + \varepsilon$$

with $\varepsilon \sim N(0, \sigma^2)$ being iid noise. Although we can easily find the distribution for $\log Y$, we would need to use the distribution of a transformation to find the distribution for $Y$. It's fairly straightforward to show the distribution is a LogNormal distribution, but if you use a more complicated transformation, the distribution may need to be derived by hand.