11
$\begingroup$

I am reading a textbook on machine learning (Data Mining by Witten, et al., 2011) and came across this passage:

... Moreover, different distributions can be used. Although the normal distribution is usually a good choice for numeric attributes, it is not suitable for attributes that have a predetermined minimum but no upper bound; in this case a "log-normal" distribution is more appropriate. Numeric attributes that are bounded above and below can be modeled by a "log-odds" distribution.

I have never heard of this distribution. I googled for "log-odds distribution" but could not find any relevant exact match. Can someone help me out? What is this distribution, and why does it help with numbers bounded above and below?

P.S. I am a software engineer, not a statistician.

$\endgroup$
14
$\begingroup$

why does it help with numbers bounded above and below?

A distribution defined on $(0,1)$ is what makes it suitable as a model for data on $(0,1)$. I don't think the text implies anything more than "it's a model for data on $(0,1)$" (or more generally, on $(a,b)$).

what is this distribution ... ?

The term 'log-odds distribution' is, unfortunately, not completely standard (and not a very common term even then).

I'll discuss some possibilities for what it might mean. Let's start by considering a way to construct distributions for values in the unit interval.

A common way to model a continuous random variable, $P$ in $(0,1)$ is the beta distribution, and a common way to model discrete proportions in $[0,1]$ is a scaled binomial ($P=X/n$, at least when $X$ is a count).

An alternative to using a beta distribution would be to take some continuous inverse CDF ($F^{-1}$) and use it to transform the values in $(0,1)$ to the real line (or rarely, the real half-line) and then use any relevant distribution ($G$) to model the values on the transformed range. This opens up many possibilities, since any pair of continuous distributions on the real line ($F,G$) are available for the transformation and the model.

So, for example, the log-odds transformation $Y=\log(\frac{P}{1-P})$ (also called the logit) would be one such inverse-cdf transformation (being the inverse CDF of a standard logistic), and then there are many distributions we might consider as models for $Y$.

We might then use (for example) a logistic$(\mu,\tau)$ model for $Y$, a simple two-parameter family on the real line. Transforming back to $(0,1)$ via the inverse log-odds transformation (i.e. $P=\frac{\exp(Y)}{1+\exp(Y)}$) yields a two parameter distribution for $P$, one that can be unimodal, or U shaped, or J shaped, symmetric or skew, in many ways somewhat like a beta distribution (personally, I'd call this logit-logistic, since its logit is logistic). Here are some examples for different values of $\mu,\tau$:

$\hspace{1.5cm}$enter image description here

Looking at the brief mention in the text by Witten et al, this might be what's intended by "log-odds distribution" - but they might as easily mean something else.

Another possibility is that the logit-normal was intended.

However, the term seems to have been used by van Erp & van Gelder (2008)$^{[1]}$, for example, to refer to a log-odds transformation on a beta distribution (so in effect taking $F$ as a logistic and $G$ as the distribution of the log of a beta-prime random variable, or equivalently the distribution of the difference of the logs of two chi-square random variables). However, they are using this to do model count proportions, which are discrete. This of course, leads to some problems (caused by trying to model a distribution with finite probability at 0 and 1 with one on $(0,1)$), which they then seem to spend a lot of effort on. (It would seem easier to just avoid the inappropriate model, but maybe that's just me.)

Several other documents (I found at least three) refer to the sample distribution of log-odds (i.e. on the scale of $Y$ above) as "the log-odds distribution" (in some cases where $P$ is a discrete proportion* and in some cases where it's a continuous proportion) - so in that case it's not a probability model as such, but it's something to which you might apply some distributional model on the real line.

* again, this has the problem that if $P$ is exactly 0 or 1, the value of $Y$ will be $-\infty$ or $\infty$ respectively ... which suggests we must bound the distribution away from 0 and 1 to use it for this purpose.

The dissertation by Yan Guo (2009)$^{[2]}$ uses the term to refer to a log-logistic distribution, a right-skew distribution on the real half-line.

So as you see, it's not a term with a single meaning. Without a clearer indication from Witten or one of the other authors of that book, we're left to guess what is intended.

[1]: Noel van Erp & Pieter van Gelder, (2008),
"How to Interpret the Beta Distribution in Case of a Breakdown,"
Proceedings of the 6th International Probabilistic Workshop, Darmstadt
pdf link

[2]: Yan Guo, (2009),
The New Methods on NDE Systems Pod Capability Assessment and Robustness,
Dissertation submitted to the Graduate School of Wayne State University, Detroit, Michigan

$\endgroup$
2
  • 1
    $\begingroup$ (+1) A search of the entire book indicates that no clarification is forthcoming. The context suggests that "log-odds distribution" refers to some particular model, just as the "lognormal" is proposed in the previous sentence as a universal distribution for all nonnegative values(!). $\endgroup$
    – whuber
    Oct 27 '14 at 17:44
  • 1
    $\begingroup$ @whuber I agree with your characterization of what's in the book - I didn't intend that my comments relating to the use of the term in other contexts to refer to the sample distribution imply that that was the intent in the book, but only as an indication of it being a term with several meanings. On the passages in question, my advice to people learning this material (as on many things) would be to read more than one book. $\endgroup$
    – Glen_b
    Oct 27 '14 at 20:35
2
$\begingroup$

I'm a software engineer (not a statistician) and I recently read a book called An Introduction to Statistical Learning. With applications in R.

I think what you're reading about is log-odds or logit. page 132

http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Fourth%20Printing.pdf

Brilliant book - I read it from cover to cover. Hope this helps

$\endgroup$
3
  • $\begingroup$ Thank you for the pointer. Assuming the log-odds distribution is the same as "logistic distribution", I looked up the latter on Wikipedia. It appears that its PDF has no lower or upper bound. So I'm still wondering why the textbook that I quoted originally said that "Numeric attributes that are bounded above and below are can be modeled" with this distribution. $\endgroup$ Oct 26 '14 at 0:36
  • $\begingroup$ I think its maybe talking about the output of the function where the bounds are 0.0 (impossible) to 1.0 (definite). (I could be completely wrong here) $\endgroup$ Oct 26 '14 at 0:46
  • $\begingroup$ It is possible that your model could produce arbitrarily large positive or negative results. These might not be interpretable in terms of a bounded range such as a probability, but could be interpretable as log-odds using the logit function and its inverse the logistic function. $\endgroup$
    – Henry
    Oct 27 '14 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.