5
$\begingroup$

I’m taking an online “Intro to AI” course for which I’m doing some azure machine learning labs. This course is largely about how to apply azure ML solutions and, while there is an “essential math for ML module”, it doesn’t do any sort of deep dive into the math/statistics. But I want to understand what’s happening mathematically to a basic level. (I've taken an intro to statistics, that's it so far.) Below are some details about some specific things I’d like to understand more, and my specific questions.

At a particular point in the “training a classification model” lab, there is this text: “The distribution of the Age column in the diabetes.csv dataset is skewed because most patients are in the youngest age bracket. Creating a version of this feature that uses a natural log transformation can help create a more linear relationship between Age and other features, and improve the ability to predict the Diabetic label. This kind of feature engineering as it’s called is common in machine learning data preparation.”

Then the instructions show you how to use Azure ML Studio to apply the natural log operation to the age column in the data set: natural log transform in azure ml studio

And after doing that, the data includes the original/raw age data and the transformed age data: Resulting data visualized

Question #1: What is that transformation actually doing? I don't mean the nitty gritty math, but what is it doing conceptually?

Question #2: The next general question is around why is a transformation necessary. On that, I did some research and found this article (https://www.r-statistics.com/2013/05/log-transformations-for-skewed-and-wide-distributions-from-practical-data-science-with-r/) that describes which type of log transformations to use in a few scenarios. Here’s a snip of text from the article: “The need for data transformation can depend on the modeling method that you plan to use. For linear and logistic regression, for example, you ideally want to make sure that the relationship between input variables and output variables is approximately linear, that the input variables are approximately normal in distribution, and that the output variable is constant variance (that is, the variance of the output variable is independent of the input variables). You may need to transform some of your input variables to better meet these assumptions.”

I don’t understand why any of that is necessary. I’ve broken my question down:

For linear and logistic regression, for example, you ideally want to make sure that: the relationship between input variables and output variables is approximately linear – why? the input variables are approximately normal in distribution- why? the output variable is constant variance (that is, the variance of the output variable is independent of the input variables – why?

Could someone help with information for those whys, or point me to material that would help?

$\endgroup$

marked as duplicate by whuber regression Mar 23 at 14:05

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ I hate it when questions such as yours are put on hold or marked as duplicate, as we cannot answer them properly. So we're reduced to having to just add comments instead of answers! $\endgroup$ – Isabella Ghement Mar 23 at 18:02
  • 1
    $\begingroup$ One intuitive way to think about what the log transformation is doing is to look at the distribution of Age - it's right skewed, so many of the ages are 'clumped together' at the lower end of the age scale, while fewer ages are spread out at the higher end of the age scale. When you log transform the distribution of Age, you 'stretch out' the clumped values to make it easier to see what is going on. In other words, the log transformed age values at the lower end of the scale are better separated from each other, instead of crowding each other. $\endgroup$ – Isabella Ghement Mar 23 at 18:10
  • 1
    $\begingroup$ Look at the high spike in the distribution of Age (a clear sign of clumping or crowdedness). That spike was attenuated in the distribution of the log-transformed Age. $\endgroup$ – Isabella Ghement Mar 23 at 18:13
  • 1
    $\begingroup$ Strictly speaking, transforming Age before including it in your model is not necessary. In fact, you could fit your model using the original Age variable first and then check the model diagnostics for any evidence that a transformation of Age might improve the model fit. People tend to transform predictor variables such as Age when they have skewed distrubutions because they think that would lead to an improved model. Whether or not that is the case is debatable. Additionally, transforming variables can complicate the interpretation of the model. $\endgroup$ – Isabella Ghement Mar 23 at 18:17
  • 1
    $\begingroup$ To give you a concrete example, let's say that all predictor variables in your model except for Age are continuous and have roughly symmetric distributions. You fit a linear regression model including all the predictors as they are (i.e., untransformed) to your data and look at the distribution of the model residuals. If this distribution reflects the skeweness of the Age distribution, then you'd be entitled to think that log-transforming Age prior to including it in your model would resolve the skewness of the residual distribution. $\endgroup$ – Isabella Ghement Mar 23 at 18:21
8
$\begingroup$

There's a lot here to break down. I hate to say it, but some of the advice in your course is quite misguided and wrong.

What is that transformation actually doing? I don't mean the nitty gritty math, but what is it doing conceptually?

The math here is pretty simple. You have a bunch of measurements of people's age that you would like to use as a feature in predicting some other measurement (looks like the probability of something happening). You're simply creating a new feature which is the logarithm of the original feature. I'll explain why you would want to do this below.

For linear and logistic regression, for example, you ideally want to make sure that: the relationship between input variables and output variables is approximately linear – why?

This is a structural assumption of the linear and logistic regression models. I'll focus on linear regression, because its a bit simpler, but the same thing holds for logistic regression.

The linear regression model makes predictions by building a formula based on the data you feed into the algorithm. All prediction models work this way, but linear regression is distinguished by building the simplest possible formula. If $y$ is the thing you are trying to predict, and $x_1, x_2, \ldots$ are the features you are using to predict it, then the linear regression formula is:

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k$$

Here, the $\beta_i$'s are just numbers, and the job of the algorithm is to determine what numbers work best.

Notice that if you vary one of the $x$'s, and look at how the output $y$ changes as a result, you'll get a line. This is a direct consequence of the way the linear regression model works. If you want it to give you sensible results, then you need to make sure this drawing lines assumption is at least approximately true.

the input variables are approximately normal in distribution- why?

This is simply wrong. Linear regression works fine even if the distribution of the input variables is highly non-normal. What is important is the relationship between the inputs and outputs, not the distribution of the inputs themselves.

This is what I meant by the advice the course being misguided. You don't transform input variables because their distribution is skew, you transform them so that the linear shape the model is trying to draw through your data is reasonable

For example, here is a scatterplot I found online of a country's GDP vs. its average life expectancy (attribution is in the image):

GDP vs. Life Expectancy

Clearly, drawing a line through the scatter plot is completely unreasonable, so the linear regression equation:

$$ \text{Life Expectancy} = \beta_0 + \beta_1 \text{GDP} $$

is a bad choice for the data. On the other hand, it looks like a logarithmic relationship is reasonable, so something like:

$$ \text{Life Expectancy} = \beta_0 + \beta_1 \log(\text{GDP}) $$

looks like it would work a lot better. This is the type of situation where transforming the GDP measurements with a logarithm is a good idea. But it has nothing to do with the distribution of GDP. You can't tell it's a good idea by drawing a histogram of GDP, it's about the relationship between GDP and life expectancy.

the output variable is constant variance (that is, the variance of the output variable is independent of the input variables – why?

This is a deeper issue of a different nature than the others. For prediction models, it doesn't really matter, so if you're focusing on learning to build good predictive models don't worry about it for now.

As a summary, this assumption is intended to support the computation of the sampling distribution of parameter estimates. For example, if you want to say something like "the probability that I would collect data in which the relationship between log(GDP) and Life Expectancy is greater than what I actually observed, even when the there is truly no relationship, is very, very small" you need to be able to compute the sampling distribution of the parameter estimates. There are various assumptions that allow this to be done, and this constant variance assumption is one them.

That said, if you're only trying to make predictions, this isn't really relevant. And in no case is the distribution of the input data assumed to be normal, that's just a misconception.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.